Sep 16 18:40:44.074454 ip-10-0-140-67 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:44.440143 ip-10-0-140-67 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:44.440143 ip-10-0-140-67 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:44.440143 ip-10-0-140-67 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:44.440143 ip-10-0-140-67 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:44.440143 ip-10-0-140-67 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:44.441821 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.441731    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:44.446096 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446067    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:44.446096 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446092    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:44.446096 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446097    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:44.446096 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446100    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:44.446096 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446103    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446120    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446123    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446126    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446129    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446131    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446134    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446137    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446139    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446142    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446144    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446147    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446150    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446152    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446155    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446157    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446161    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446166    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446168    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:44.446318 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446171    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446174    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446176    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446180    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446184    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446191    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446193    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446196    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446199    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446201    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446204    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446207    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446210    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446212    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446215    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446218    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446221    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446225    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446228    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:44.446785 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446231    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446234    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446237    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446240    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446243    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446245    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446248    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446251    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446253    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446256    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446259    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446261    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446265    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446267    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446269    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446272    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446274    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446277    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446280    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446282    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:44.447276 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446285    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446287    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446290    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446293    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446296    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446299    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446302    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446304    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446309    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446312    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446314    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446317    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446320    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446322    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446325    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446327    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446330    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446333    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446336    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446338    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:44.447750 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446341    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446344    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446347    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446349    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446352    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446354    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446357    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446360    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446363    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446365    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446368    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446370    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446373    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446375    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446378    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446381    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.446383    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447080    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447087    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:44.448243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447090    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447093    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447096    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447099    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447101    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447104    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447122    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447125    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447130    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447134    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447137    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447141    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447143    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447146    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447149    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447152    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447154    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447157    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447160    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:44.448699 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447162    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447165    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447168    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447170    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447173    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447176    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447178    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447181    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447183    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447186    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447189    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447192    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447195    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447197    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447200    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447203    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447206    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447208    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447211    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447214    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:44.449182 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447216    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447219    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447221    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447224    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447227    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447230    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447232    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447235    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447238    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447240    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447243    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447245    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447248    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447250    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447252    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447255    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447257    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447260    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447262    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447265    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:44.449669 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447267    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447270    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447272    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447275    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447278    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447280    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447284    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447288    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447291    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447294    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447296    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447299    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447301    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447304    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447306    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447309    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447311    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447314    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447316    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:44.450203 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447319    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447323    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447327    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447330    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447334    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447337    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447340    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447343    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447345    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447348    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447350    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447353    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447355    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447358    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447361    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447364    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447366    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447369    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447372    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:44.450667 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447774    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447779    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447782    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447784    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447787    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447790    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447792    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447795    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447798    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447800    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447802    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447805    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447807    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447810    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447813    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447815    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447818    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447821    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447826    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:44.451127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447830    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447833    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447836    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447839    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447841    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447844    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447848    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447852    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447855    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447857    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447860    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447862    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447865    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447868    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447871    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447873    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447875    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447878    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447880    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:44.451632 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447883    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447885    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447888    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447890    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447892    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447895    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447898    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447901    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447904    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447906    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447909    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447912    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447914    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447917    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447919    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447922    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447925    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447927    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447930    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447932    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:44.452092 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447935    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447938    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447940    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447943    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447946    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447950    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447953    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447955    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447958    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447960    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447963    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447965    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447968    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447970    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447973    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447975    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447978    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447980    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447982    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447985    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:44.452595 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447987    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447989    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447992    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447994    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.447997    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448000    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448002    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448005    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448007    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448010    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448012    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448014    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448017    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448019    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448022    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448024    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448027    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448029    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448032    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448034    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:44.453127 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.448037    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448130    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448181    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448188    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448193    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448197    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448200    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448204    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448209    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448212    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448215    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448219    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448222    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448225    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448228    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448231    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448234    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448238    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448241    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448245    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448248    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448251    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448254    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448257    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:44.453610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448262    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448265    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448268    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448271    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448274    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448277    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448283    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448286    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448289    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448293    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448297    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448300    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448302    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448305    2579 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448308    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448964    2579 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448968    2579 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448970    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448974    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448977    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448981    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448984    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448987    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448990    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448993    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:44.454185 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.448996    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449001    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449004    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449007    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449010    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449012    2579 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449016    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449021    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449024    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449028    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449031    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449034    2579 flags.go:64] FLAG: --help="false"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449037    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449041    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449044    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449047    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449050    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449054    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449057    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449060    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449062    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449065    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449068    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449071    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:44.454781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449074    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449077    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449080    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449083    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449086    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449089    2579 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449091    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449094    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449097    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449102    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449124    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449129    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449134    2579 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449138    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449142    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449146    2579 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449149    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449153    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449157    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449161    2579 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449164    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449168    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449171    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449174    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449178    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:44.455384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449181    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449184    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449193    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449196    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449199    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449202    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449205    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449210    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449213    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449216    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449219    2579 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449222    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449225    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-033496cb5a50da7c4"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449229    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449232    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449235    2579 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449238    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449246    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449250    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449253    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449256    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449259    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449263    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449266    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449269    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:44.455984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449272    2579 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449275    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449279    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449283    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449286    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449289    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449292    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449295    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449298    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449301    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449304    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449306    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449309    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449312    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449315    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449318    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449323    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449326    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449329    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449332    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449335    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449338    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449341    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449343    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449346    2579 flags.go:64] FLAG: --v="2"
Sep 16 18:40:44.456661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449350    2579 flags.go:64] FLAG: --version="false"
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449364    2579 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449368    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.449371    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449501    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449505    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449509    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449512    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449514    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449517    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449521    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449523    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449526    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449528    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449531    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449533    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449535    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449539    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449543    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:44.457402 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449546    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449550    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449553    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449556    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449559    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449562    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449565    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449567    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449570    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449572    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449575    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449577    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449579    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449582    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449585    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449588    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449591    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449593    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449596    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449598    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:44.457871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449601    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449603    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449606    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449609    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449612    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449614    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449617    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449619    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449622    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449625    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449627    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449630    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449632    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449635    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449637    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449639    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449642    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449645    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449647    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449649    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:44.458373 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449652    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449654    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449657    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449659    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449662    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449664    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449667    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449669    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449672    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449675    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449678    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449680    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449683    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449685    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449688    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449692    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449694    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449698    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449701    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449704    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:44.458871 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449707    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449710    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449712    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449715    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449717    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449719    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449722    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449724    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449727    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449729    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449732    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449734    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449737    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449740    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449742    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449744    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449749    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449753    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449755    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:44.459369 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449758    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449760    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449764    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449766    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.449769    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.450504    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.458149    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.458168    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.458190    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.458199    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458257    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458262    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458265    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458268    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458271    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:44.459827 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458274    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458277    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458280    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458283    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458286    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458289    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458292    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458295    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458297    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458300    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458303    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458306    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458308    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458311    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458314    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458316    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458319    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458321    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458324    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458328    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:44.460242 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458330    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458333    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458335    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458338    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458341    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458344    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458346    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458349    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458352    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458355    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458357    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458360    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458363    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458365    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458368    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458370    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458373    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458376    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458379    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:44.460736 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458381    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458384    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458387    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458390    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458392    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458395    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458397    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458400    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458404    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458409    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458414    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458417    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458421    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458425    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458427    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458430    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458432    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458436    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:44.461217 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458439    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458441    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458445    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458447    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458450    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458452    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458455    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458457    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458460    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458463    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458465    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458468    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458470    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458473    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458475    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458478    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458481    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458483    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458486    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458488    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:44.461649 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458491    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458493    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458496    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458499    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458501    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458504    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458506    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458509    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458512    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458515    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458517    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458520    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458523    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458526    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458529    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458531    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:44.462220 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458533    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.458538    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458651    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458656    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458659    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458662    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458664    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458667    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458670    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458673    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458676    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458678    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458681    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458683    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:44.462594 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458686    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458689    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458691    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458693    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458696    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458699    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458702    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458704    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458707    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458709    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458711    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458714    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458719    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458722    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458725    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458728    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458730    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458733    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458735    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:44.462936 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458738    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458740    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458743    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458745    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458747    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458750    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458752    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458754    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458757    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458759    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458762    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458764    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458766    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458769    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458771    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458773    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458776    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458778    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458781    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458784    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:44.463409 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458786    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458789    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458791    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458793    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458796    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458799    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458801    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458804    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458806    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458809    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458811    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458814    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458816    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458818    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458821    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458823    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458825    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458828    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458830    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458833    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:44.463869 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458835    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458837    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458840    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458843    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458845    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458847    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458850    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458852    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458854    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458857    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458859    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458862    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458864    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458867    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458870    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458872    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458875    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458878    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458883    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:44.464384 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458886    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458889    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458891    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458894    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458896    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458899    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458901    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458904    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:44.458906    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.458911    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.459033    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.461089    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.461938    2579 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.462039    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:44.464861 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.462075    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:44.465224 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.464344    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:44.465512 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.465493    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:44.474615 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.474593    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:44.477904 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.477882    2579 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:44.478776 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.478754    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:44.483252 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.483231    2579 fs.go:135] Filesystem UUIDs: map[5cbd0c0e-f666-4cb9-9239-8aa6c1315dff:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 de832be0-2f70-4140-b280-413b3b269c1d:/dev/nvme0n1p3]
Sep 16 18:40:44.483324 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.483251    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:44.489177 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.489039    2579 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:44.487846645 +0000 UTC m=+0.303159254 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113278 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fd98b1548d185cdd4fbeff5ebb60b SystemUUID:ec2fd98b-1548-d185-cdd4-fbeff5ebb60b BootID:77791c5a-3d5c-4e3d-94a2-9942973cdadd Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:b6:da:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:b6:da:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:82:05:a6:b3:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:44.489177 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.489173    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:44.489293 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.489260    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:44.490378 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.490355    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:44.490519 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.490381    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-67.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:44.491152 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.491142    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:44.491187 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.491155    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:44.491187 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.491164    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:44.491760 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.491748    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:44.492612 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.492601    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:44.495895 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.495882    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:44.496011 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.496002    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:44.496057 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.496033    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:44.496057 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.496045    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:44.496142 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.496060    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:44.496142 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.496069    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:44.496142 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.496078    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:44.496535 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.496520    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:44.496570 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.496544    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:44.497881 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.497859    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:44.499866 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.499839    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:44.502321 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.502306    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:44.502390 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.502337    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:44.503875 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503856    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:44.503875 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503875    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:44.503998 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503882    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:44.503998 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503887    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:44.503998 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503901    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:44.503998 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503909    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:44.503998 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503928    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:44.503998 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503934    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:44.503998 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503941    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:44.503998 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503947    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:44.503998 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503955    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:44.503998 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.503962    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:44.504740 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.504729    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:44.504740 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.504740    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:44.506122 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.506089    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:44.506207 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.506164    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:44.508816 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.508802    2579 server.go:1268] "Started kubelet"
Sep 16 18:40:44.508909 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.508881    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:44.509007 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.508947    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:44.509102 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.509026    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:44.510266 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.510251    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:44.512626 ip-10-0-140-67 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:44.512833 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.512763    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-67.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:44.514655 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.514634    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:44.515841 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.515702    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:44.516744 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.516729    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:44.516798 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.516752    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:44.516877 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.516853    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:44.517381 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.517358    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:44.517650 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.517633    2579 factory.go:55] Registering systemd factory
Sep 16 18:40:44.517738 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.517654    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:44.517885 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.517872    2579 factory.go:153] Registering CRI-O factory
Sep 16 18:40:44.517965 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.517889    2579 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:44.517965 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.517941    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:44.517965 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.517961    2579 factory.go:103] Registering Raw factory
Sep 16 18:40:44.518097 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.517976    2579 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:44.518097 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.517974    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:44.518097 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.518034    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:44.518248 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.518100    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:44.518248 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.518131    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:44.518387 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.518374    2579 manager.go:349] Starting recovery of all containers
Sep 16 18:40:44.526600 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.525872    2579 manager.go:354] Recovery completed
Sep 16 18:40:44.528741 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.528711    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:44.528865 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.528814    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:44.529869 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.528713    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-67.ec2.internal.18d5e134f4a6e75f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-67.ec2.internal,UID:ip-10-0-140-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-67.ec2.internal,},FirstTimestamp:2026-09-16 18:40:44.508768095 +0000 UTC m=+0.324080744,LastTimestamp:2026-09-16 18:40:44.508768095 +0000 UTC m=+0.324080744,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-67.ec2.internal,}"
Sep 16 18:40:44.530852 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.530830    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:44.532670 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.532651    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:44.532728 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.532688    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:44.532728 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.532702    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:44.533360 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.533346    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:44.533360 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.533358    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:44.533464 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.533365    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:44.533464 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.533374    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:44.534223 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.534192    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:44.534287 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.534223    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:44.534287 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.534237    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:44.535069 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.535004    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-67.ec2.internal.18d5e134f613a0e4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-67.ec2.internal,UID:ip-10-0-140-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-67.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-67.ec2.internal,},FirstTimestamp:2026-09-16 18:40:44.532670692 +0000 UTC m=+0.347983288,LastTimestamp:2026-09-16 18:40:44.532670692 +0000 UTC m=+0.347983288,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-67.ec2.internal,}"
Sep 16 18:40:44.535254 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.535242    2579 policy_none.go:49] "None policy: Start"
Sep 16 18:40:44.535287 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.535272    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:44.535331 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.535286    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:44.536942 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.536930    2579 policy_none.go:47] "Start"
Sep 16 18:40:44.551323 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.551194    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-67.ec2.internal.18d5e134f613ff30  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-67.ec2.internal,UID:ip-10-0-140-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-67.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-67.ec2.internal,},FirstTimestamp:2026-09-16 18:40:44.532694832 +0000 UTC m=+0.348007426,LastTimestamp:2026-09-16 18:40:44.532694832 +0000 UTC m=+0.348007426,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-67.ec2.internal,}"
Sep 16 18:40:44.553013 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.552947    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-67.ec2.internal.18d5e134f6142fd3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-67.ec2.internal,UID:ip-10-0-140-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-67.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-67.ec2.internal,},FirstTimestamp:2026-09-16 18:40:44.532707283 +0000 UTC m=+0.348019877,LastTimestamp:2026-09-16 18:40:44.532707283 +0000 UTC m=+0.348019877,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-67.ec2.internal,}"
Sep 16 18:40:44.564528 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.564424    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-67.ec2.internal.18d5e134f613a0e4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-67.ec2.internal.18d5e134f613a0e4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-67.ec2.internal,UID:ip-10-0-140-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-67.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-67.ec2.internal,},FirstTimestamp:2026-09-16 18:40:44.532670692 +0000 UTC m=+0.347983288,LastTimestamp:2026-09-16 18:40:44.534206123 +0000 UTC m=+0.349518715,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-67.ec2.internal,}"
Sep 16 18:40:44.571982 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.571912    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-67.ec2.internal.18d5e134f613ff30\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-67.ec2.internal.18d5e134f613ff30  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-67.ec2.internal,UID:ip-10-0-140-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-67.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-67.ec2.internal,},FirstTimestamp:2026-09-16 18:40:44.532694832 +0000 UTC m=+0.348007426,LastTimestamp:2026-09-16 18:40:44.534229305 +0000 UTC m=+0.349541896,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-67.ec2.internal,}"
Sep 16 18:40:44.575683 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.575652    2579 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.575712    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.575727    2579 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.576069    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.576080    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.576189    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.576296    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.576304    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.576955    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q7xmg"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.576978    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.577023    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.584085    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-67.ec2.internal.18d5e134f6142fd3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-67.ec2.internal.18d5e134f6142fd3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-67.ec2.internal,UID:ip-10-0-140-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-67.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-67.ec2.internal,},FirstTimestamp:2026-09-16 18:40:44.532707283 +0000 UTC m=+0.348019877,LastTimestamp:2026-09-16 18:40:44.534242928 +0000 UTC m=+0.349555518,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-67.ec2.internal,}"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.591626    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q7xmg"
Sep 16 18:40:44.601523 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.598084    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-67.ec2.internal.18d5e134f8c8ac33  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-67.ec2.internal,UID:ip-10-0-140-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-140-67.ec2.internal,},FirstTimestamp:2026-09-16 18:40:44.578090035 +0000 UTC m=+0.393402611,LastTimestamp:2026-09-16 18:40:44.578090035 +0000 UTC m=+0.393402611,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-67.ec2.internal,}"
Sep 16 18:40:44.654027 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.653979    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:44.655391 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.655370    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:44.655495 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.655431    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:44.655495 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.655463    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:44.655587 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.655509    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:44.670923 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.670891    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:44.676763 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.676745    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:44.677762 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.677742    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:44.677861 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.677780    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:44.677861 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.677795    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:44.677861 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.677823    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.688992 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.688970    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.689072 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.688996    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-67.ec2.internal\": node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:44.729506 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.729420    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:44.755828 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.755791    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal"]
Sep 16 18:40:44.755930 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.755915    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:44.757129 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.757096    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:44.757219 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.757149    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:44.757219 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.757165    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:44.758728 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.758714    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:44.758850 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.758833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.758890 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.758866    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:44.759651 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.759635    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:44.759733 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.759659    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:44.759733 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.759667    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:44.759733 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.759681    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:44.759733 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.759682    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:44.759733 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.759730    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:44.761468 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.761447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.761553 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.761478    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:44.762822 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.762805    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:44.762905 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.762828    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:44.762905 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.762837    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:44.785091 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.785067    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-67.ec2.internal\" not found" node="ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.789436 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.789420    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-67.ec2.internal\" not found" node="ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.819746 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.819720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/412608350c6496727b5c4e59f9f387b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal\" (UID: \"412608350c6496727b5c4e59f9f387b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.819835 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.819750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/412608350c6496727b5c4e59f9f387b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal\" (UID: \"412608350c6496727b5c4e59f9f387b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.819835 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.819779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a79a00728263f7068c7da6157868c572-config\") pod \"kube-apiserver-proxy-ip-10-0-140-67.ec2.internal\" (UID: \"a79a00728263f7068c7da6157868c572\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.829840 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.829819    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:44.920562 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.920528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/412608350c6496727b5c4e59f9f387b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal\" (UID: \"412608350c6496727b5c4e59f9f387b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.920562 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.920561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/412608350c6496727b5c4e59f9f387b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal\" (UID: \"412608350c6496727b5c4e59f9f387b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.920753 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.920579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a79a00728263f7068c7da6157868c572-config\") pod \"kube-apiserver-proxy-ip-10-0-140-67.ec2.internal\" (UID: \"a79a00728263f7068c7da6157868c572\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.920753 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.920629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/412608350c6496727b5c4e59f9f387b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal\" (UID: \"412608350c6496727b5c4e59f9f387b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.920753 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.920633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a79a00728263f7068c7da6157868c572-config\") pod \"kube-apiserver-proxy-ip-10-0-140-67.ec2.internal\" (UID: \"a79a00728263f7068c7da6157868c572\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.920753 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:44.920633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/412608350c6496727b5c4e59f9f387b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal\" (UID: \"412608350c6496727b5c4e59f9f387b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:44.930641 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:44.930615    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.031586 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:45.031504    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.087700 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.087673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:45.092624 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.092605    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:45.131735 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:45.131701    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.232244 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:45.232193    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.332812 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:45.332729    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.433464 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:45.433431    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.449793 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.449758    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:45.462453 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.462426    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:45.462604 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.462569    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:45.462604 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.462585    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:45.534366 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:45.534329    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.584597 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:45.584525    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412608350c6496727b5c4e59f9f387b9.slice/crio-01858cce6cab01209fe8701702336b7b9c2a3bc4680172db0927bbc9a3b085ae WatchSource:0}: Error finding container 01858cce6cab01209fe8701702336b7b9c2a3bc4680172db0927bbc9a3b085ae: Status 404 returned error can't find the container with id 01858cce6cab01209fe8701702336b7b9c2a3bc4680172db0927bbc9a3b085ae
Sep 16 18:40:45.589252 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.589233    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:45.596860 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.596817    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:45 +0000 UTC" deadline="2028-03-06 03:21:57.898119074 +0000 UTC"
Sep 16 18:40:45.596942 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.596876    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12872h41m12.301248565s"
Sep 16 18:40:45.610425 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.610392    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:45.611404 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.611382    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:45.611491 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.611418    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:45.611491 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.611432    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:45.628668 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:45.628644    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79a00728263f7068c7da6157868c572.slice/crio-2a00c2ca134ebc209f03c9fc03640edc16372588b175ffcd302dab7678e6dc27 WatchSource:0}: Error finding container 2a00c2ca134ebc209f03c9fc03640edc16372588b175ffcd302dab7678e6dc27: Status 404 returned error can't find the container with id 2a00c2ca134ebc209f03c9fc03640edc16372588b175ffcd302dab7678e6dc27
Sep 16 18:40:45.634720 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:45.634700    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.661574 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.661531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal" event={"ID":"a79a00728263f7068c7da6157868c572","Type":"ContainerStarted","Data":"2a00c2ca134ebc209f03c9fc03640edc16372588b175ffcd302dab7678e6dc27"}
Sep 16 18:40:45.662691 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.662671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal" event={"ID":"412608350c6496727b5c4e59f9f387b9","Type":"ContainerStarted","Data":"01858cce6cab01209fe8701702336b7b9c2a3bc4680172db0927bbc9a3b085ae"}
Sep 16 18:40:45.735770 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:45.735738    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.763834 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.763803    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:45.836152 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:45.836055    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.936641 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:45.936612    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-67.ec2.internal\" not found"
Sep 16 18:40:45.944774 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:45.944755    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:46.017442 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:46.017411    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:46.043226 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:46.043193    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:46.044146 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:46.044120    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal"
Sep 16 18:40:46.062319 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:46.062285    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:47.647123 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.647047    2579 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:47.655978 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.655955    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:47.656865 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.656832    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-l6fts","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal","openshift-multus/multus-additional-cni-plugins-lw7zd","openshift-multus/network-metrics-daemon-srqqp","openshift-cluster-node-tuning-operator/tuned-5f2tp","openshift-dns/node-resolver-s5dww","openshift-multus/multus-nf4w7","openshift-network-diagnostics/network-check-target-7fssr","openshift-network-operator/iptables-alerter-99nzs","openshift-ovn-kubernetes/ovnkube-node-smblz","kube-system/global-pull-secret-syncer-46l2m","kube-system/konnectivity-agent-spz42","kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"]
Sep 16 18:40:47.658468 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.658447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.659630 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.659612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:40:47.661459 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.660937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.663216 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.662482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:47.663734 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.663713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:47.663815 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.663788    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:40:47.665753 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.665727    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:47.666243 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666224    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:47.666332 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666224    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:47.666492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666473    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:47.666575 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666551    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:47.666575 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666566    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:47.666681 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666479    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:47.666763 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666742    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:47.666823 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666779    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:47.666823 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666790    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x4cbm\""
Sep 16 18:40:47.666943 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q4qr9\""
Sep 16 18:40:47.666943 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666931    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:47.667128 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.666986    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zv5rc\""
Sep 16 18:40:47.667128 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.667099    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:47.667376 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.667302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.667376 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.667327    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c8dx8\""
Sep 16 18:40:47.667497 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.667444    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:47.667849 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.667833    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:47.667996 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.667982    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:47.668175 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.668142    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:47.668489 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.668475    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m6kgg\""
Sep 16 18:40:47.668561 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.668502    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:47.668703 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.668689    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:47.669161 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.669127    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:47.669236 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.669205    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:40:47.669236 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.669229    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:47.670772 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.670753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:47.670866 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.670810    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-525qs\""
Sep 16 18:40:47.671976 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.671949    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:47.673346 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.672136    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:47.673346 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.672273    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:47.673346 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.672327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.673612 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.673598    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.674674 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.674654    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:47.674835 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.674809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:47.674914 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.674873    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:40:47.674979 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.674927    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4vt5b\""
Sep 16 18:40:47.675038 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.674990    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:47.676012 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.675983    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:47.676258 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.676240    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6f6rf\""
Sep 16 18:40:47.676372 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.676353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.676749 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.676731    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:47.679297 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679275    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:47.679403 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q8q7\" (UniqueName: \"kubernetes.io/projected/9465c804-604d-44af-be5d-523b881029e8-kube-api-access-9q8q7\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prt4p\" (UniqueName: \"kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p\") pod \"network-check-target-7fssr\" (UID: \"664b3dff-845a-40e3-a987-8ff3d897d228\") " pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-etc-openvswitch\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-kubernetes\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-sys\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679572    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lnhvl\""
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e38c60c-611a-4f3e-99c7-5911dcf78a33-tmp\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-multus-cni-dir\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-sysctl-d\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d9a4c61-28b6-4ab1-88c0-4d431578e06f-hosts-file\") pod \"node-resolver-s5dww\" (UID: \"9d9a4c61-28b6-4ab1-88c0-4d431578e06f\") " pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-hostroot\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pgd\" (UniqueName: \"kubernetes.io/projected/7e38c60c-611a-4f3e-99c7-5911dcf78a33-kube-api-access-62pgd\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjmgl\" (UniqueName: \"kubernetes.io/projected/ac00e65a-d284-4f21-86d0-50dd1b532515-kube-api-access-jjmgl\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-run-k8s-cni-cncf-io\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-run-netns\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-slash\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.680105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac00e65a-d284-4f21-86d0-50dd1b532515-env-overrides\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-run-systemd\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvml\" (UniqueName: \"kubernetes.io/projected/8a2b3c86-57c3-4cd2-8142-5be14426c108-kube-api-access-kkvml\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-var-lib-kubelet\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.679996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-node-log\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac00e65a-d284-4f21-86d0-50dd1b532515-ovnkube-script-lib\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ea0010d-1f3d-40ee-8b69-787ecf68bf84-konnectivity-ca\") pod \"konnectivity-agent-spz42\" (UID: \"4ea0010d-1f3d-40ee-8b69-787ecf68bf84\") " pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-var-lib-openvswitch\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-cni-bin\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac00e65a-d284-4f21-86d0-50dd1b532515-ovn-node-metrics-cert\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-run-multus-certs\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-var-lib-kubelet\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac00e65a-d284-4f21-86d0-50dd1b532515-ovnkube-config\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9a4c61-28b6-4ab1-88c0-4d431578e06f-tmp-dir\") pod \"node-resolver-s5dww\" (UID: \"9d9a4c61-28b6-4ab1-88c0-4d431578e06f\") " pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680322    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-var-lib-cni-multus\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-cni-netd\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681014 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-cnibin\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-lib-modules\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-host\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-multus-conf-dir\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.680615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvt42\" (UniqueName: \"kubernetes.io/projected/da6bcad0-944e-4ba5-99c7-b0867431ed13-kube-api-access-gvt42\") pod \"iptables-alerter-99nzs\" (UID: \"da6bcad0-944e-4ba5-99c7-b0867431ed13\") " pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.681059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-systemd-units\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.681101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da6bcad0-944e-4ba5-99c7-b0867431ed13-iptables-alerter-script\") pod \"iptables-alerter-99nzs\" (UID: \"da6bcad0-944e-4ba5-99c7-b0867431ed13\") " pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.681139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da6bcad0-944e-4ba5-99c7-b0867431ed13-host-slash\") pod \"iptables-alerter-99nzs\" (UID: \"da6bcad0-944e-4ba5-99c7-b0867431ed13\") " pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.681196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d65d4d93-e2cc-4ced-b860-e6c1accf08b7-serviceca\") pod \"node-ca-l6fts\" (UID: \"d65d4d93-e2cc-4ced-b860-e6c1accf08b7\") " pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.681225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-system-cni-dir\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.681249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9465c804-604d-44af-be5d-523b881029e8-cni-binary-copy\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.681282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-run-netns\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.681934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.681311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87xsj\" (UniqueName: \"kubernetes.io/projected/d65d4d93-e2cc-4ced-b860-e6c1accf08b7-kube-api-access-87xsj\") pod \"node-ca-l6fts\" (UID: \"d65d4d93-e2cc-4ced-b860-e6c1accf08b7\") " pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.681991    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-etc-kubernetes\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-log-socket\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-run-ovn-kubernetes\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-tuned\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-modprobe-d\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-sysconfig\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-sysctl-conf\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-systemd\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-run-openvswitch\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-run-ovn\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ea0010d-1f3d-40ee-8b69-787ecf68bf84-agent-certs\") pod \"konnectivity-agent-spz42\" (UID: \"4ea0010d-1f3d-40ee-8b69-787ecf68bf84\") " pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-multus-socket-dir-parent\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-run\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-kubelet\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.682580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682574    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.683429 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65d4d93-e2cc-4ced-b860-e6c1accf08b7-host\") pod \"node-ca-l6fts\" (UID: \"d65d4d93-e2cc-4ced-b860-e6c1accf08b7\") " pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:47.683429 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsv79\" (UniqueName: \"kubernetes.io/projected/9d9a4c61-28b6-4ab1-88c0-4d431578e06f-kube-api-access-wsv79\") pod \"node-resolver-s5dww\" (UID: \"9d9a4c61-28b6-4ab1-88c0-4d431578e06f\") " pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:47.683429 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-var-lib-cni-bin\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.683429 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.682702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9465c804-604d-44af-be5d-523b881029e8-multus-daemon-config\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.768814 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.768783    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:47.780852 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.780822    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n6866"
Sep 16 18:40:47.782957 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.782937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmgl\" (UniqueName: \"kubernetes.io/projected/ac00e65a-d284-4f21-86d0-50dd1b532515-kube-api-access-jjmgl\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783063 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.782968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-run-k8s-cni-cncf-io\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.783063 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.782990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-run-netns\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.783063 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.783209 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-socket-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.783209 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-run-netns\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.783209 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-slash\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783209 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac00e65a-d284-4f21-86d0-50dd1b532515-env-overrides\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783209 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-run-k8s-cni-cncf-io\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.783421 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-slash\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783421 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-registration-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.783421 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-run-systemd\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783421 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvml\" (UniqueName: \"kubernetes.io/projected/8a2b3c86-57c3-4cd2-8142-5be14426c108-kube-api-access-kkvml\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:47.783421 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-var-lib-kubelet\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.783421 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-node-log\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783421 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-run-systemd\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac00e65a-d284-4f21-86d0-50dd1b532515-ovnkube-script-lib\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-var-lib-kubelet\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-node-log\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ea0010d-1f3d-40ee-8b69-787ecf68bf84-konnectivity-ca\") pod \"konnectivity-agent-spz42\" (UID: \"4ea0010d-1f3d-40ee-8b69-787ecf68bf84\") " pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-cnibin\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-sys-fs\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-var-lib-openvswitch\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-cni-bin\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac00e65a-d284-4f21-86d0-50dd1b532515-ovn-node-metrics-cert\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-var-lib-openvswitch\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-cni-bin\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-run-multus-certs\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-var-lib-kubelet\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.783741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b52d6a7-170d-47a1-b771-ff55013146d6-dbus\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-run-multus-certs\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac00e65a-d284-4f21-86d0-50dd1b532515-ovnkube-config\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac00e65a-d284-4f21-86d0-50dd1b532515-env-overrides\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-var-lib-kubelet\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9a4c61-28b6-4ab1-88c0-4d431578e06f-tmp-dir\") pod \"node-resolver-s5dww\" (UID: \"9d9a4c61-28b6-4ab1-88c0-4d431578e06f\") " pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.783904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-var-lib-cni-multus\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-cni-netd\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac00e65a-d284-4f21-86d0-50dd1b532515-ovnkube-script-lib\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-var-lib-cni-multus\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ea0010d-1f3d-40ee-8b69-787ecf68bf84-konnectivity-ca\") pod \"konnectivity-agent-spz42\" (UID: \"4ea0010d-1f3d-40ee-8b69-787ecf68bf84\") " pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-cnibin\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-cni-netd\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-cnibin\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.784143    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-lib-modules\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-host\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.784446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9a4c61-28b6-4ab1-88c0-4d431578e06f-tmp-dir\") pod \"node-resolver-s5dww\" (UID: \"9d9a4c61-28b6-4ab1-88c0-4d431578e06f\") " pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.784226    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs podName:8a2b3c86-57c3-4cd2-8142-5be14426c108 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:48.284205154 +0000 UTC m=+2.949477578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs") pod "network-metrics-daemon-srqqp" (UID: "8a2b3c86-57c3-4cd2-8142-5be14426c108") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac00e65a-d284-4f21-86d0-50dd1b532515-ovnkube-config\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-multus-conf-dir\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvt42\" (UniqueName: \"kubernetes.io/projected/da6bcad0-944e-4ba5-99c7-b0867431ed13-kube-api-access-gvt42\") pod \"iptables-alerter-99nzs\" (UID: \"da6bcad0-944e-4ba5-99c7-b0867431ed13\") " pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-multus-conf-dir\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-host\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-systemd-units\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-systemd-units\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-lib-modules\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da6bcad0-944e-4ba5-99c7-b0867431ed13-iptables-alerter-script\") pod \"iptables-alerter-99nzs\" (UID: \"da6bcad0-944e-4ba5-99c7-b0867431ed13\") " pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da6bcad0-944e-4ba5-99c7-b0867431ed13-host-slash\") pod \"iptables-alerter-99nzs\" (UID: \"da6bcad0-944e-4ba5-99c7-b0867431ed13\") " pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d65d4d93-e2cc-4ced-b860-e6c1accf08b7-serviceca\") pod \"node-ca-l6fts\" (UID: \"d65d4d93-e2cc-4ced-b860-e6c1accf08b7\") " pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-system-cni-dir\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784405    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9465c804-604d-44af-be5d-523b881029e8-cni-binary-copy\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-run-netns\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.785290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da6bcad0-944e-4ba5-99c7-b0867431ed13-host-slash\") pod \"iptables-alerter-99nzs\" (UID: \"da6bcad0-944e-4ba5-99c7-b0867431ed13\") " pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87xsj\" (UniqueName: \"kubernetes.io/projected/d65d4d93-e2cc-4ced-b860-e6c1accf08b7-kube-api-access-87xsj\") pod \"node-ca-l6fts\" (UID: \"d65d4d93-e2cc-4ced-b860-e6c1accf08b7\") " pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784522    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-system-cni-dir\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-etc-kubernetes\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-run-netns\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-system-cni-dir\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-etc-kubernetes\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvs5c\" (UniqueName: \"kubernetes.io/projected/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-kube-api-access-xvs5c\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb7q9\" (UniqueName: \"kubernetes.io/projected/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-kube-api-access-xb7q9\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-log-socket\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-run-ovn-kubernetes\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-tuned\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-run-ovn-kubernetes\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-log-socket\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.786065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-modprobe-d\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-sysconfig\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-sysctl-conf\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d65d4d93-e2cc-4ced-b860-e6c1accf08b7-serviceca\") pod \"node-ca-l6fts\" (UID: \"d65d4d93-e2cc-4ced-b860-e6c1accf08b7\") " pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-systemd\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b52d6a7-170d-47a1-b771-ff55013146d6-kubelet-config\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784968    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da6bcad0-944e-4ba5-99c7-b0867431ed13-iptables-alerter-script\") pod \"iptables-alerter-99nzs\" (UID: \"da6bcad0-944e-4ba5-99c7-b0867431ed13\") " pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-run-openvswitch\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-systemd\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-run-ovn\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.784998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-sysctl-conf\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9465c804-604d-44af-be5d-523b881029e8-cni-binary-copy\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ea0010d-1f3d-40ee-8b69-787ecf68bf84-agent-certs\") pod \"konnectivity-agent-spz42\" (UID: \"4ea0010d-1f3d-40ee-8b69-787ecf68bf84\") " pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-run-openvswitch\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-sysconfig\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-multus-socket-dir-parent\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-modprobe-d\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-run-ovn\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.786887 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-run\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-device-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-multus-socket-dir-parent\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-run\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-etc-selinux\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-kubelet\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65d4d93-e2cc-4ced-b860-e6c1accf08b7-host\") pod \"node-ca-l6fts\" (UID: \"d65d4d93-e2cc-4ced-b860-e6c1accf08b7\") " pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-kubelet\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsv79\" (UniqueName: \"kubernetes.io/projected/9d9a4c61-28b6-4ab1-88c0-4d431578e06f-kube-api-access-wsv79\") pod \"node-resolver-s5dww\" (UID: \"9d9a4c61-28b6-4ab1-88c0-4d431578e06f\") " pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-var-lib-cni-bin\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65d4d93-e2cc-4ced-b860-e6c1accf08b7-host\") pod \"node-ca-l6fts\" (UID: \"d65d4d93-e2cc-4ced-b860-e6c1accf08b7\") " pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9465c804-604d-44af-be5d-523b881029e8-multus-daemon-config\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.785798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-host-var-lib-cni-bin\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q8q7\" (UniqueName: \"kubernetes.io/projected/9465c804-604d-44af-be5d-523b881029e8-kube-api-access-9q8q7\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.787723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt4p\" (UniqueName: \"kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p\") pod \"network-check-target-7fssr\" (UID: \"664b3dff-845a-40e3-a987-8ff3d897d228\") " pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-etc-openvswitch\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-kubernetes\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-sys\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e38c60c-611a-4f3e-99c7-5911dcf78a33-tmp\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-multus-cni-dir\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-sysctl-d\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-kubernetes\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d9a4c61-28b6-4ab1-88c0-4d431578e06f-hosts-file\") pod \"node-resolver-s5dww\" (UID: \"9d9a4c61-28b6-4ab1-88c0-4d431578e06f\") " pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-hostroot\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pgd\" (UniqueName: \"kubernetes.io/projected/7e38c60c-611a-4f3e-99c7-5911dcf78a33-kube-api-access-62pgd\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d9a4c61-28b6-4ab1-88c0-4d431578e06f-hosts-file\") pod \"node-resolver-s5dww\" (UID: \"9d9a4c61-28b6-4ab1-88c0-4d431578e06f\") " pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-sys\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac00e65a-d284-4f21-86d0-50dd1b532515-etc-openvswitch\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9465c804-604d-44af-be5d-523b881029e8-multus-daemon-config\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-multus-cni-dir\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9465c804-604d-44af-be5d-523b881029e8-hostroot\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.788490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.786709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-sysctl-d\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.789181 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.788940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e38c60c-611a-4f3e-99c7-5911dcf78a33-etc-tuned\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.789181 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.789136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e38c60c-611a-4f3e-99c7-5911dcf78a33-tmp\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.789342 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.789298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac00e65a-d284-4f21-86d0-50dd1b532515-ovn-node-metrics-cert\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.789430 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.789399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ea0010d-1f3d-40ee-8b69-787ecf68bf84-agent-certs\") pod \"konnectivity-agent-spz42\" (UID: \"4ea0010d-1f3d-40ee-8b69-787ecf68bf84\") " pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:40:47.797388 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.797371    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n6866"
Sep 16 18:40:47.800417 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.800396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmgl\" (UniqueName: \"kubernetes.io/projected/ac00e65a-d284-4f21-86d0-50dd1b532515-kube-api-access-jjmgl\") pod \"ovnkube-node-smblz\" (UID: \"ac00e65a-d284-4f21-86d0-50dd1b532515\") " pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:47.800962 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.800943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvml\" (UniqueName: \"kubernetes.io/projected/8a2b3c86-57c3-4cd2-8142-5be14426c108-kube-api-access-kkvml\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:47.804476 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.804452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvt42\" (UniqueName: \"kubernetes.io/projected/da6bcad0-944e-4ba5-99c7-b0867431ed13-kube-api-access-gvt42\") pod \"iptables-alerter-99nzs\" (UID: \"da6bcad0-944e-4ba5-99c7-b0867431ed13\") " pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.804830 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.804813    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:47.804830 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.804831    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:47.804954 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.804843    2579 projected.go:196] Error preparing data for projected volume kube-api-access-prt4p for pod openshift-network-diagnostics/network-check-target-7fssr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:47.804954 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.804901    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p podName:664b3dff-845a-40e3-a987-8ff3d897d228 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:48.304889058 +0000 UTC m=+2.970161469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-prt4p" (UniqueName: "kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p") pod "network-check-target-7fssr" (UID: "664b3dff-845a-40e3-a987-8ff3d897d228") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:47.807979 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.807918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q8q7\" (UniqueName: \"kubernetes.io/projected/9465c804-604d-44af-be5d-523b881029e8-kube-api-access-9q8q7\") pod \"multus-nf4w7\" (UID: \"9465c804-604d-44af-be5d-523b881029e8\") " pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:47.810007 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.809988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87xsj\" (UniqueName: \"kubernetes.io/projected/d65d4d93-e2cc-4ced-b860-e6c1accf08b7-kube-api-access-87xsj\") pod \"node-ca-l6fts\" (UID: \"d65d4d93-e2cc-4ced-b860-e6c1accf08b7\") " pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:47.810144 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.810007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsv79\" (UniqueName: \"kubernetes.io/projected/9d9a4c61-28b6-4ab1-88c0-4d431578e06f-kube-api-access-wsv79\") pod \"node-resolver-s5dww\" (UID: \"9d9a4c61-28b6-4ab1-88c0-4d431578e06f\") " pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:47.810347 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.810331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pgd\" (UniqueName: \"kubernetes.io/projected/7e38c60c-611a-4f3e-99c7-5911dcf78a33-kube-api-access-62pgd\") pod \"tuned-5f2tp\" (UID: \"7e38c60c-611a-4f3e-99c7-5911dcf78a33\") " pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:47.887123 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b52d6a7-170d-47a1-b771-ff55013146d6-kubelet-config\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:47.887123 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.887382 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-device-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.887382 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-etc-selinux\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.887382 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-device-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.887382 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b52d6a7-170d-47a1-b771-ff55013146d6-kubelet-config\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:47.887382 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.887629 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-socket-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.887629 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887410    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-etc-selinux\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.887629 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-registration-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.887629 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:47.887629 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-cnibin\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.887629 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-sys-fs\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.887629 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b52d6a7-170d-47a1-b771-ff55013146d6-dbus\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:47.887629 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-system-cni-dir\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvs5c\" (UniqueName: \"kubernetes.io/projected/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-kube-api-access-xvs5c\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb7q9\" (UniqueName: \"kubernetes.io/projected/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-kube-api-access-xb7q9\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b52d6a7-170d-47a1-b771-ff55013146d6-dbus\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-cnibin\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.887987 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.887956    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:47.888468 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.888000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-sys-fs\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.888468 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:47.888013    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret podName:5b52d6a7-170d-47a1-b771-ff55013146d6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:48.387996998 +0000 UTC m=+3.053269408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret") pod "global-pull-secret-syncer-46l2m" (UID: "5b52d6a7-170d-47a1-b771-ff55013146d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:47.888468 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.888070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-registration-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.888468 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.887960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-socket-dir\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.888468 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.888168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-system-cni-dir\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.888468 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.888217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.888468 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.888416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.904905 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.904841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb7q9\" (UniqueName: \"kubernetes.io/projected/7f01d77e-33de-4985-8d37-7dd1f7a4fd46-kube-api-access-xb7q9\") pod \"aws-ebs-csi-driver-node-244rj\" (UID: \"7f01d77e-33de-4985-8d37-7dd1f7a4fd46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:47.906440 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.906420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvs5c\" (UniqueName: \"kubernetes.io/projected/4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b-kube-api-access-xvs5c\") pod \"multus-additional-cni-plugins-lw7zd\" (UID: \"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b\") " pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:47.976199 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.976167    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-99nzs"
Sep 16 18:40:47.985325 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.985302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:40:47.993138 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:47.993117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:40:48.000372 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.000353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l6fts"
Sep 16 18:40:48.006097 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.006073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s5dww"
Sep 16 18:40:48.013714 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.013694    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nf4w7"
Sep 16 18:40:48.020385 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.020364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5f2tp"
Sep 16 18:40:48.027114 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.027096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lw7zd"
Sep 16 18:40:48.032632 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.032615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj"
Sep 16 18:40:48.075411 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.075380    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:48.271171 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.271080    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:48.289722 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.289689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:48.289857 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:48.289834    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:48.289909 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:48.289899    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs podName:8a2b3c86-57c3-4cd2-8142-5be14426c108 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:49.289885305 +0000 UTC m=+3.955157716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs") pod "network-metrics-daemon-srqqp" (UID: "8a2b3c86-57c3-4cd2-8142-5be14426c108") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:48.391043 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.391004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:48.391261 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.391110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt4p\" (UniqueName: \"kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p\") pod \"network-check-target-7fssr\" (UID: \"664b3dff-845a-40e3-a987-8ff3d897d228\") " pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:48.391261 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:48.391212    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:48.391392 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:48.391271    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:48.391392 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:48.391293    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:48.391392 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:48.391301    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret podName:5b52d6a7-170d-47a1-b771-ff55013146d6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:49.39128039 +0000 UTC m=+4.056552814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret") pod "global-pull-secret-syncer-46l2m" (UID: "5b52d6a7-170d-47a1-b771-ff55013146d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:48.391392 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:48.391311    2579 projected.go:196] Error preparing data for projected volume kube-api-access-prt4p for pod openshift-network-diagnostics/network-check-target-7fssr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:48.391392 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:48.391374    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p podName:664b3dff-845a-40e3-a987-8ff3d897d228 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:49.391358475 +0000 UTC m=+4.056630885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-prt4p" (UniqueName: "kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p") pod "network-check-target-7fssr" (UID: "664b3dff-845a-40e3-a987-8ff3d897d228") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:48.662622 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:48.662597    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f01d77e_33de_4985_8d37_7dd1f7a4fd46.slice/crio-6c006d66509a5dd85d0adca7e4899e74270f73aef26dc616a26564112f41d52c WatchSource:0}: Error finding container 6c006d66509a5dd85d0adca7e4899e74270f73aef26dc616a26564112f41d52c: Status 404 returned error can't find the container with id 6c006d66509a5dd85d0adca7e4899e74270f73aef26dc616a26564112f41d52c
Sep 16 18:40:48.664634 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:48.664606    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9465c804_604d_44af_be5d_523b881029e8.slice/crio-8a46d21993399c545aa4cb79b61da74631d56783956ebc9f51ab0875e37cd79b WatchSource:0}: Error finding container 8a46d21993399c545aa4cb79b61da74631d56783956ebc9f51ab0875e37cd79b: Status 404 returned error can't find the container with id 8a46d21993399c545aa4cb79b61da74631d56783956ebc9f51ab0875e37cd79b
Sep 16 18:40:48.665986 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:48.665962    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65d4d93_e2cc_4ced_b860_e6c1accf08b7.slice/crio-c4f4dcfb81a5554fcae818016f6d51ed3d9bbe8dff11e9dc386932b0d093847d WatchSource:0}: Error finding container c4f4dcfb81a5554fcae818016f6d51ed3d9bbe8dff11e9dc386932b0d093847d: Status 404 returned error can't find the container with id c4f4dcfb81a5554fcae818016f6d51ed3d9bbe8dff11e9dc386932b0d093847d
Sep 16 18:40:48.668466 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:48.667765    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9a4c61_28b6_4ab1_88c0_4d431578e06f.slice/crio-f84d4b11cb8f3ce5a3d93c65745901f1308a0b52cb35d4d97bfaa9318a485967 WatchSource:0}: Error finding container f84d4b11cb8f3ce5a3d93c65745901f1308a0b52cb35d4d97bfaa9318a485967: Status 404 returned error can't find the container with id f84d4b11cb8f3ce5a3d93c65745901f1308a0b52cb35d4d97bfaa9318a485967
Sep 16 18:40:48.668680 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:48.668656    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4125e54e_d784_4ac0_b8b6_c18c5f6d6a5b.slice/crio-14e15a68221a279603e109016ddb10dfdc51de805960c26d27182c2dcc4b0bd5 WatchSource:0}: Error finding container 14e15a68221a279603e109016ddb10dfdc51de805960c26d27182c2dcc4b0bd5: Status 404 returned error can't find the container with id 14e15a68221a279603e109016ddb10dfdc51de805960c26d27182c2dcc4b0bd5
Sep 16 18:40:48.670503 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:48.670486    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac00e65a_d284_4f21_86d0_50dd1b532515.slice/crio-45763d3063ed142d6045f15271b30e193918c88325ebd9534203fdb5615ed223 WatchSource:0}: Error finding container 45763d3063ed142d6045f15271b30e193918c88325ebd9534203fdb5615ed223: Status 404 returned error can't find the container with id 45763d3063ed142d6045f15271b30e193918c88325ebd9534203fdb5615ed223
Sep 16 18:40:48.671532 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:48.671425    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6bcad0_944e_4ba5_99c7_b0867431ed13.slice/crio-4d220b7f5bbd0e2058cc1b509633f7435aa438fb27c30a58690292751f61f491 WatchSource:0}: Error finding container 4d220b7f5bbd0e2058cc1b509633f7435aa438fb27c30a58690292751f61f491: Status 404 returned error can't find the container with id 4d220b7f5bbd0e2058cc1b509633f7435aa438fb27c30a58690292751f61f491
Sep 16 18:40:48.672223 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:48.672097    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e38c60c_611a_4f3e_99c7_5911dcf78a33.slice/crio-7ad7b83aa3dab8c336fcc044ac70636f71395601ae36c7214df4ca0889586f00 WatchSource:0}: Error finding container 7ad7b83aa3dab8c336fcc044ac70636f71395601ae36c7214df4ca0889586f00: Status 404 returned error can't find the container with id 7ad7b83aa3dab8c336fcc044ac70636f71395601ae36c7214df4ca0889586f00
Sep 16 18:40:48.674081 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:40:48.674065    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea0010d_1f3d_40ee_8b69_787ecf68bf84.slice/crio-35358d38bde1fd168f5737c140ef2fc8df369ed237ada656e15a8168163b990d WatchSource:0}: Error finding container 35358d38bde1fd168f5737c140ef2fc8df369ed237ada656e15a8168163b990d: Status 404 returned error can't find the container with id 35358d38bde1fd168f5737c140ef2fc8df369ed237ada656e15a8168163b990d
Sep 16 18:40:48.800571 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.800526    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:47 +0000 UTC" deadline="2028-05-10 11:26:36.760439427 +0000 UTC"
Sep 16 18:40:48.800571 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.800567    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14440h45m47.959876936s"
Sep 16 18:40:48.816895 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.816860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-99nzs" event={"ID":"da6bcad0-944e-4ba5-99c7-b0867431ed13","Type":"ContainerStarted","Data":"4d220b7f5bbd0e2058cc1b509633f7435aa438fb27c30a58690292751f61f491"}
Sep 16 18:40:48.818124 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.818101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw7zd" event={"ID":"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b","Type":"ContainerStarted","Data":"14e15a68221a279603e109016ddb10dfdc51de805960c26d27182c2dcc4b0bd5"}
Sep 16 18:40:48.819181 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.819134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l6fts" event={"ID":"d65d4d93-e2cc-4ced-b860-e6c1accf08b7","Type":"ContainerStarted","Data":"c4f4dcfb81a5554fcae818016f6d51ed3d9bbe8dff11e9dc386932b0d093847d"}
Sep 16 18:40:48.821068 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.821045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal" event={"ID":"a79a00728263f7068c7da6157868c572","Type":"ContainerStarted","Data":"00d94f235c7b48807a18586ca9b0df483cbbf7893189e184fd64ecc52ce16ad2"}
Sep 16 18:40:48.822246 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.822225    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5f2tp" event={"ID":"7e38c60c-611a-4f3e-99c7-5911dcf78a33","Type":"ContainerStarted","Data":"7ad7b83aa3dab8c336fcc044ac70636f71395601ae36c7214df4ca0889586f00"}
Sep 16 18:40:48.823523 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.823495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smblz" event={"ID":"ac00e65a-d284-4f21-86d0-50dd1b532515","Type":"ContainerStarted","Data":"45763d3063ed142d6045f15271b30e193918c88325ebd9534203fdb5615ed223"}
Sep 16 18:40:48.824780 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.824757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s5dww" event={"ID":"9d9a4c61-28b6-4ab1-88c0-4d431578e06f","Type":"ContainerStarted","Data":"f84d4b11cb8f3ce5a3d93c65745901f1308a0b52cb35d4d97bfaa9318a485967"}
Sep 16 18:40:48.825815 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.825797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nf4w7" event={"ID":"9465c804-604d-44af-be5d-523b881029e8","Type":"ContainerStarted","Data":"8a46d21993399c545aa4cb79b61da74631d56783956ebc9f51ab0875e37cd79b"}
Sep 16 18:40:48.826954 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.826927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj" event={"ID":"7f01d77e-33de-4985-8d37-7dd1f7a4fd46","Type":"ContainerStarted","Data":"6c006d66509a5dd85d0adca7e4899e74270f73aef26dc616a26564112f41d52c"}
Sep 16 18:40:48.827943 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:48.827923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-spz42" event={"ID":"4ea0010d-1f3d-40ee-8b69-787ecf68bf84","Type":"ContainerStarted","Data":"35358d38bde1fd168f5737c140ef2fc8df369ed237ada656e15a8168163b990d"}
Sep 16 18:40:49.297807 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.297721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:49.297973 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.297831    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:49.297973 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.297880    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs podName:8a2b3c86-57c3-4cd2-8142-5be14426c108 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:51.2978669 +0000 UTC m=+5.963139308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs") pod "network-metrics-daemon-srqqp" (UID: "8a2b3c86-57c3-4cd2-8142-5be14426c108") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:49.398265 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.398225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:49.398455 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.398333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt4p\" (UniqueName: \"kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p\") pod \"network-check-target-7fssr\" (UID: \"664b3dff-845a-40e3-a987-8ff3d897d228\") " pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:49.398552 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.398525    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:49.398609 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.398563    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:49.398609 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.398576    2579 projected.go:196] Error preparing data for projected volume kube-api-access-prt4p for pod openshift-network-diagnostics/network-check-target-7fssr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:49.398708 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.398637    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p podName:664b3dff-845a-40e3-a987-8ff3d897d228 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:51.398617074 +0000 UTC m=+6.063889488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-prt4p" (UniqueName: "kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p") pod "network-check-target-7fssr" (UID: "664b3dff-845a-40e3-a987-8ff3d897d228") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:49.399068 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.399051    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:49.399137 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.399106    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret podName:5b52d6a7-170d-47a1-b771-ff55013146d6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:51.399091214 +0000 UTC m=+6.064363628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret") pod "global-pull-secret-syncer-46l2m" (UID: "5b52d6a7-170d-47a1-b771-ff55013146d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:49.801767 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.801698    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:47 +0000 UTC" deadline="2028-02-19 20:11:14.005681839 +0000 UTC"
Sep 16 18:40:49.801767 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.801762    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12505h30m24.203923836s"
Sep 16 18:40:49.808425 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.808397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:49.808543 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.808519    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:40:49.808907 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.808892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:49.809011 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.808993    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:40:49.809068 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.809055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:49.809184 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:49.809129    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:40:49.837053 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.836470    2579 generic.go:356] "Generic (PLEG): container finished" podID="412608350c6496727b5c4e59f9f387b9" containerID="6d370d7dc9fd5d16735af02caa943963b71dc7eae8713ee7987667b68400ed82" exitCode=0
Sep 16 18:40:49.837053 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.836551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal" event={"ID":"412608350c6496727b5c4e59f9f387b9","Type":"ContainerDied","Data":"6d370d7dc9fd5d16735af02caa943963b71dc7eae8713ee7987667b68400ed82"}
Sep 16 18:40:49.862266 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:49.862211    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-67.ec2.internal" podStartSLOduration=2.862189522 podStartE2EDuration="2.862189522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:48.851107271 +0000 UTC m=+3.516379721" watchObservedRunningTime="2026-09-16 18:40:49.862189522 +0000 UTC m=+4.527461956"
Sep 16 18:40:50.856556 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:50.855565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal" event={"ID":"412608350c6496727b5c4e59f9f387b9","Type":"ContainerStarted","Data":"cf2ec321ae92c9abe3bec582d24f6045f76a71296233702fa26a3139ba1b09ce"}
Sep 16 18:40:51.316505 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:51.316419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:51.316660 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.316627    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:51.316722 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.316691    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs podName:8a2b3c86-57c3-4cd2-8142-5be14426c108 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:55.316673362 +0000 UTC m=+9.981945773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs") pod "network-metrics-daemon-srqqp" (UID: "8a2b3c86-57c3-4cd2-8142-5be14426c108") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:51.416841 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:51.416800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:51.417017 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:51.416900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt4p\" (UniqueName: \"kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p\") pod \"network-check-target-7fssr\" (UID: \"664b3dff-845a-40e3-a987-8ff3d897d228\") " pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:51.417073 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.417050    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:51.417073 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.417070    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:51.417218 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.417083    2579 projected.go:196] Error preparing data for projected volume kube-api-access-prt4p for pod openshift-network-diagnostics/network-check-target-7fssr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:51.417218 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.417139    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p podName:664b3dff-845a-40e3-a987-8ff3d897d228 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:55.417120076 +0000 UTC m=+10.082392489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-prt4p" (UniqueName: "kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p") pod "network-check-target-7fssr" (UID: "664b3dff-845a-40e3-a987-8ff3d897d228") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:51.417570 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.417550    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:51.417654 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.417608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret podName:5b52d6a7-170d-47a1-b771-ff55013146d6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:55.417593516 +0000 UTC m=+10.082865932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret") pod "global-pull-secret-syncer-46l2m" (UID: "5b52d6a7-170d-47a1-b771-ff55013146d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:51.807075 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:51.806255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:51.807075 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.806390    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:40:51.807075 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:51.806781    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:51.807075 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.806869    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:40:51.807075 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:51.806919    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:51.807075 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:51.806989    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:40:53.807559 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:53.807492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:53.808064 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:53.807640    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:40:53.808199 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:53.808177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:53.808326 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:53.808281    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:40:53.808464 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:53.808447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:53.808568 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:53.808550    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:40:55.353289 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:55.353250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:55.353774 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.353410    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:55.353774 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.353464    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs podName:8a2b3c86-57c3-4cd2-8142-5be14426c108 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:03.353450724 +0000 UTC m=+18.018723133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs") pod "network-metrics-daemon-srqqp" (UID: "8a2b3c86-57c3-4cd2-8142-5be14426c108") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:55.453794 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:55.453752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt4p\" (UniqueName: \"kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p\") pod \"network-check-target-7fssr\" (UID: \"664b3dff-845a-40e3-a987-8ff3d897d228\") " pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:55.453976 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:55.453817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:55.453976 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.453943    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:55.453976 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.453972    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:55.454094 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.453989    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:55.454094 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.454004    2579 projected.go:196] Error preparing data for projected volume kube-api-access-prt4p for pod openshift-network-diagnostics/network-check-target-7fssr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:55.454094 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.454004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret podName:5b52d6a7-170d-47a1-b771-ff55013146d6 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:03.453987818 +0000 UTC m=+18.119260244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret") pod "global-pull-secret-syncer-46l2m" (UID: "5b52d6a7-170d-47a1-b771-ff55013146d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:55.454094 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.454060    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p podName:664b3dff-845a-40e3-a987-8ff3d897d228 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:03.454049913 +0000 UTC m=+18.119322321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-prt4p" (UniqueName: "kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p") pod "network-check-target-7fssr" (UID: "664b3dff-845a-40e3-a987-8ff3d897d228") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:55.807308 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:55.807237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:55.807435 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:55.807335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:55.807473 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.807432    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:40:55.807473 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.807333    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:40:55.807473 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:55.807235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:55.807565 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:55.807513    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:40:57.805734 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:57.805701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:57.805734 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:57.805720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:57.806249 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:57.805834    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:40:57.806249 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:57.805913    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:40:57.806249 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:57.805951    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:57.806249 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:57.806037    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:40:59.806740 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:59.806655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:40:59.806740 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:59.806698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:40:59.807248 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:40:59.806657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:40:59.807248 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:59.806802    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:40:59.807248 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:59.806874    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:40:59.807248 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:40:59.806936    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:01.806682 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:01.806643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:01.806682 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:01.806687    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:01.807197 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:01.806794    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:01.807197 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:01.806885    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:01.807197 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:01.806921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:01.807197 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:01.807019    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:03.415036 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:03.414998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:03.415500 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.415126    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:03.415500 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.415211    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs podName:8a2b3c86-57c3-4cd2-8142-5be14426c108 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:19.415190438 +0000 UTC m=+34.080463041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs") pod "network-metrics-daemon-srqqp" (UID: "8a2b3c86-57c3-4cd2-8142-5be14426c108") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:03.515822 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:03.515787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:03.515992 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:03.515866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt4p\" (UniqueName: \"kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p\") pod \"network-check-target-7fssr\" (UID: \"664b3dff-845a-40e3-a987-8ff3d897d228\") " pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:03.515992 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.515959    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:03.515992 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.515979    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:41:03.516117 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.515996    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:41:03.516117 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.516008    2579 projected.go:196] Error preparing data for projected volume kube-api-access-prt4p for pod openshift-network-diagnostics/network-check-target-7fssr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:03.516117 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.516035    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret podName:5b52d6a7-170d-47a1-b771-ff55013146d6 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:19.516014293 +0000 UTC m=+34.181286715 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret") pod "global-pull-secret-syncer-46l2m" (UID: "5b52d6a7-170d-47a1-b771-ff55013146d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:03.516117 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.516058    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p podName:664b3dff-845a-40e3-a987-8ff3d897d228 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:19.516042606 +0000 UTC m=+34.181315030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-prt4p" (UniqueName: "kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p") pod "network-check-target-7fssr" (UID: "664b3dff-845a-40e3-a987-8ff3d897d228") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:03.806776 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:03.806698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:03.806938 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:03.806698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:03.806938 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.806822    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:03.806938 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:03.806700    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:03.806938 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.806907    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:03.807144 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:03.806997    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:05.807244 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:05.807198    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:05.807716 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:05.807299    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:05.807716 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:05.807333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:05.807716 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:05.807405    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:05.807716 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:05.807420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:05.807716 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:05.807482    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:06.895463 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:06.895433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5f2tp" event={"ID":"7e38c60c-611a-4f3e-99c7-5911dcf78a33","Type":"ContainerStarted","Data":"b23b650d506e91e600ff4b78522f59c1b95bce26b6c10df1bfd16b0219619144"}
Sep 16 18:41:06.899874 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:06.899826    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nf4w7" event={"ID":"9465c804-604d-44af-be5d-523b881029e8","Type":"ContainerStarted","Data":"6fd87579e6d82ba41192787c8bbb19e59baef62ca5a2cf76da297d37f95fd894"}
Sep 16 18:41:06.931681 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:06.930594    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-67.ec2.internal" podStartSLOduration=19.930570208 podStartE2EDuration="19.930570208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:50.876104767 +0000 UTC m=+5.541377213" watchObservedRunningTime="2026-09-16 18:41:06.930570208 +0000 UTC m=+21.595842638"
Sep 16 18:41:06.968961 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:06.968606    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5f2tp" podStartSLOduration=2.9310228929999997 podStartE2EDuration="20.968582817s" totalImagesPullingTime="18.037559924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:06.93132414 +0000 UTC m=+21.596596572" watchObservedRunningTime="2026-09-16 18:41:06.968582817 +0000 UTC m=+21.633855251"
Sep 16 18:41:06.969089 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:06.968974    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nf4w7" podStartSLOduration=2.861120031 podStartE2EDuration="20.968964398s" totalImagesPullingTime="18.107844367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:06.968895478 +0000 UTC m=+21.634167894" watchObservedRunningTime="2026-09-16 18:41:06.968964398 +0000 UTC m=+21.634236830"
Sep 16 18:41:07.806747 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.806715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:07.806949 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.806722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:07.806949 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:07.806819    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:07.806949 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.806730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:07.806949 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:07.806881    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:07.807191 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:07.806954    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:07.908470 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.908432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-spz42" event={"ID":"4ea0010d-1f3d-40ee-8b69-787ecf68bf84","Type":"ContainerStarted","Data":"4af8d58db2f36bb931ae20b11d8222edb3a2eb34f4fc16041bd6e67aba7c3844"}
Sep 16 18:41:07.910143 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.910123    2579 generic.go:356] "Generic (PLEG): container finished" podID="4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b" containerID="45d61725c2688e84fb8d42e096462499c407f695917d2ff220082fb5c41b7b97" exitCode=0
Sep 16 18:41:07.910245 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.910185    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw7zd" event={"ID":"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b","Type":"ContainerDied","Data":"45d61725c2688e84fb8d42e096462499c407f695917d2ff220082fb5c41b7b97"}
Sep 16 18:41:07.911817 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.911795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l6fts" event={"ID":"d65d4d93-e2cc-4ced-b860-e6c1accf08b7","Type":"ContainerStarted","Data":"5169a8238c51184c973393bd75a5c46bb0b93298ed864c9b56d78ece8db8622c"}
Sep 16 18:41:07.914754 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.914735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smblz" event={"ID":"ac00e65a-d284-4f21-86d0-50dd1b532515","Type":"ContainerStarted","Data":"62952be0bba958534d77669af7199f4050b65994bb02f84f128282dfd5e441f7"}
Sep 16 18:41:07.914844 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.914800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smblz" event={"ID":"ac00e65a-d284-4f21-86d0-50dd1b532515","Type":"ContainerStarted","Data":"08958526eeef81d4c518b616e4ed4e348d58ee02f8309b9ec7cc933d8a1919b4"}
Sep 16 18:41:07.914844 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.914830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smblz" event={"ID":"ac00e65a-d284-4f21-86d0-50dd1b532515","Type":"ContainerStarted","Data":"327edee87a195973d042045af34be6c0fec0db6cd1d0f49be6653f0b5b9f82a7"}
Sep 16 18:41:07.914920 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.914869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smblz" event={"ID":"ac00e65a-d284-4f21-86d0-50dd1b532515","Type":"ContainerStarted","Data":"80bd7eede971f883a01d1ae4d34bc854ce0e2ca3f0b413787b28dc68f32a3e6a"}
Sep 16 18:41:07.914920 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.914899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smblz" event={"ID":"ac00e65a-d284-4f21-86d0-50dd1b532515","Type":"ContainerStarted","Data":"4bbe2acc88829d4fd4b4cda510d2234f849ab5a0ab02812a5de1c9c9ce9338f5"}
Sep 16 18:41:07.914994 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.914936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smblz" event={"ID":"ac00e65a-d284-4f21-86d0-50dd1b532515","Type":"ContainerStarted","Data":"243268ef40c363cb412d9d017c7bdcbea8517e586d497a86182dea51605d8026"}
Sep 16 18:41:07.916073 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.916049    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s5dww" event={"ID":"9d9a4c61-28b6-4ab1-88c0-4d431578e06f","Type":"ContainerStarted","Data":"c44aadd5d93baebe18ca1d1be9ef7d08a6bf754fea1cddccf1ad41254bec200b"}
Sep 16 18:41:07.917370 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.917351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj" event={"ID":"7f01d77e-33de-4985-8d37-7dd1f7a4fd46","Type":"ContainerStarted","Data":"860876c1eba2ca82864e6f9ac2536eaadc029001f694622422d726b0fe6a568a"}
Sep 16 18:41:07.935483 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.935440    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-spz42" podStartSLOduration=8.889419496 podStartE2EDuration="22.935428712s" totalImagesPullingTime="14.046009216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:07.935143922 +0000 UTC m=+22.600416377" watchObservedRunningTime="2026-09-16 18:41:07.935428712 +0000 UTC m=+22.600701141"
Sep 16 18:41:07.989577 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.989506    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l6fts" podStartSLOduration=4.946053567 podStartE2EDuration="22.989491816s" totalImagesPullingTime="18.043438249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:07.958176213 +0000 UTC m=+22.623448672" watchObservedRunningTime="2026-09-16 18:41:07.989491816 +0000 UTC m=+22.654764224"
Sep 16 18:41:07.989793 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:07.989772    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s5dww" podStartSLOduration=3.95163812 podStartE2EDuration="21.989763498s" totalImagesPullingTime="18.038125378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:07.989625451 +0000 UTC m=+22.654897892" watchObservedRunningTime="2026-09-16 18:41:07.989763498 +0000 UTC m=+22.655035928"
Sep 16 18:41:08.145294 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:08.145264    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:41:08.356206 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:08.356098    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:08.553747 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:08.553716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:08.739865 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:08.739654    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:41:08.145288543Z","UUID":"9fe8215c-73b8-4ef7-a0cc-f8864560f18f","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:41:08.741526 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:08.741499    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:41:08.741718 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:08.741539    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:41:08.922625 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:08.922583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj" event={"ID":"7f01d77e-33de-4985-8d37-7dd1f7a4fd46","Type":"ContainerStarted","Data":"aa1ee19eef3275204393c6d44aea63605cf64d559d24cffe3466a41c81efc569"}
Sep 16 18:41:08.925174 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:08.925132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-99nzs" event={"ID":"da6bcad0-944e-4ba5-99c7-b0867431ed13","Type":"ContainerStarted","Data":"56561e5e6d19e41e8296ab2e037dc4fe7131fae7fc84b8356ea4769d50dfd257"}
Sep 16 18:41:09.050335 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:09.050249    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.308694 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:09.308665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.696220 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:09.696130    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.806360 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:09.806327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:09.806360 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:09.806360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:09.806607 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:09.806360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:09.806607 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:09.806438    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:09.806607 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:09.806537    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:09.806607 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:09.806592    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:09.875601 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:09.875550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.930989 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:09.930954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smblz" event={"ID":"ac00e65a-d284-4f21-86d0-50dd1b532515","Type":"ContainerStarted","Data":"13fcebd92d3caf07494ad26612221070d7c753ef6c23d9f7b06376bdc07aafdd"}
Sep 16 18:41:09.933640 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:09.933613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj" event={"ID":"7f01d77e-33de-4985-8d37-7dd1f7a4fd46","Type":"ContainerStarted","Data":"bc9e883b6c0d1cf9a899b4797dcb6be572a0f7984c83a0cae8a52c9f3c9e43ea"}
Sep 16 18:41:09.985387 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:09.985302    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-99nzs" podStartSLOduration=6.947910265 podStartE2EDuration="24.985286584s" totalImagesPullingTime="18.037376319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:08.973576781 +0000 UTC m=+23.638849239" watchObservedRunningTime="2026-09-16 18:41:09.985286584 +0000 UTC m=+24.650559014"
Sep 16 18:41:10.013829 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:10.013805    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s5dww_9d9a4c61-28b6-4ab1-88c0-4d431578e06f/dns-node-resolver/0.log"
Sep 16 18:41:10.613763 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:10.613734    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l6fts_d65d4d93-e2cc-4ced-b860-e6c1accf08b7/node-ca/0.log"
Sep 16 18:41:10.743570 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:10.743539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:11.651967 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:11.651932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:11.806480 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:11.806402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:11.806480 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:11.806424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:11.806710 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:11.806402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:11.806710 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:11.806541    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:11.806710 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:11.806596    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:11.806710 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:11.806666    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:11.862446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:11.862412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:12.179082 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.178991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:41:12.179673 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.179653    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:41:12.208475 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.208433    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-244rj" podStartSLOduration=6.017270407 podStartE2EDuration="26.20841656s" totalImagesPullingTime="20.191146153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:09.988507696 +0000 UTC m=+24.653780131" watchObservedRunningTime="2026-09-16 18:41:12.20841656 +0000 UTC m=+26.873688987"
Sep 16 18:41:12.944376 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.944339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smblz" event={"ID":"ac00e65a-d284-4f21-86d0-50dd1b532515","Type":"ContainerStarted","Data":"e871d1cb3b97674021a3473170e698e65be6a7e8905952bcc1911ba488d49c06"}
Sep 16 18:41:12.945134 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.944570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:41:12.945134 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.944640    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:41:12.946249 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.946224    2579 generic.go:356] "Generic (PLEG): container finished" podID="4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b" containerID="7b24dbff3bf7cd44d9b06736a126f855e61439fa30af7284e299f121d902f285" exitCode=0
Sep 16 18:41:12.946344 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.946294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw7zd" event={"ID":"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b","Type":"ContainerDied","Data":"7b24dbff3bf7cd44d9b06736a126f855e61439fa30af7284e299f121d902f285"}
Sep 16 18:41:12.946526 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.946503    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:41:12.946982 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.946965    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-spz42"
Sep 16 18:41:12.959437 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.959414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:12.961205 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:12.961187    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:41:13.023485 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:13.023438    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-smblz" podStartSLOduration=9.931997328 podStartE2EDuration="28.023426583s" totalImagesPullingTime="18.091429255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:13.022654564 +0000 UTC m=+27.687927007" watchObservedRunningTime="2026-09-16 18:41:13.023426583 +0000 UTC m=+27.688699012"
Sep 16 18:41:13.267116 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:13.267013    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:13.805690 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:13.805652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:13.805845 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:13.805781    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:13.805845 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:13.805801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:13.805965 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:13.805888    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:13.805965 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:13.805940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:13.806055 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:13.806002    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:13.951308 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:13.951276    2579 generic.go:356] "Generic (PLEG): container finished" podID="4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b" containerID="1ad3b746f373f097c31c0a69fd5195f954fe6fc9614459c6e663b70a7b6df43e" exitCode=0
Sep 16 18:41:13.951907 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:13.951358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw7zd" event={"ID":"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b","Type":"ContainerDied","Data":"1ad3b746f373f097c31c0a69fd5195f954fe6fc9614459c6e663b70a7b6df43e"}
Sep 16 18:41:13.951976 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:13.951967    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:41:13.971375 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:13.971098    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:41:14.203841 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:14.203734    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-46l2m"]
Sep 16 18:41:14.203973 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:14.203863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:14.203973 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:14.203954    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:14.206825 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:14.206799    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-srqqp"]
Sep 16 18:41:14.206933 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:14.206902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:14.207010 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:14.206994    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:14.243783 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:14.243752    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7fssr"]
Sep 16 18:41:14.243919 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:14.243873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:14.244036 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:14.244011    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:14.955674 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:14.955646    2579 generic.go:356] "Generic (PLEG): container finished" podID="4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b" containerID="16894c13e71b52b1e86c399dbee65d340c3ac2b4974a192686e5bd52b7aa65a1" exitCode=0
Sep 16 18:41:14.956122 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:14.955724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw7zd" event={"ID":"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b","Type":"ContainerDied","Data":"16894c13e71b52b1e86c399dbee65d340c3ac2b4974a192686e5bd52b7aa65a1"}
Sep 16 18:41:14.982364 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:14.982342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:15.293937 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:15.293853    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:15.591956 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:15.591921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:15.763426 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:15.763356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:15.807236 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:15.807204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:15.807412 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:15.807308    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:15.807638 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:15.807611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:15.807755 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:15.807684    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:15.807755 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:15.807720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:15.807832 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:15.807782    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:16.449305 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:16.449275    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:16.825814 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:16.825782    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:17.015128 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:17.015094    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:17.730353 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:17.730317    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:17.806772 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:17.806728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:17.806941 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:17.806728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:17.806941 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:17.806867    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:17.806941 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:17.806899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:17.807064 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:17.806973    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:17.807064 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:17.807057    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:18.009935 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:18.009854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:18.388661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:18.388624    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:18.665795 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:18.665677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:18.847840 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:18.847808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:19.130767 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.130728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:19.395006 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.394941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:19.431053 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.431020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:19.431196 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.431178    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:19.431249 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.431241    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs podName:8a2b3c86-57c3-4cd2-8142-5be14426c108 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:51.431226233 +0000 UTC m=+66.096498646 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs") pod "network-metrics-daemon-srqqp" (UID: "8a2b3c86-57c3-4cd2-8142-5be14426c108") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:19.531572 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.531533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:19.531755 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.531617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt4p\" (UniqueName: \"kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p\") pod \"network-check-target-7fssr\" (UID: \"664b3dff-845a-40e3-a987-8ff3d897d228\") " pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:19.531755 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.531682    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:19.531755 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.531738    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:41:19.531755 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.531749    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret podName:5b52d6a7-170d-47a1-b771-ff55013146d6 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:51.531734045 +0000 UTC m=+66.197006456 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret") pod "global-pull-secret-syncer-46l2m" (UID: "5b52d6a7-170d-47a1-b771-ff55013146d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:19.531755 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.531755    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:41:19.531958 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.531767    2579 projected.go:196] Error preparing data for projected volume kube-api-access-prt4p for pod openshift-network-diagnostics/network-check-target-7fssr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:19.531958 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.531816    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p podName:664b3dff-845a-40e3-a987-8ff3d897d228 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:51.531798601 +0000 UTC m=+66.197071029 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-prt4p" (UniqueName: "kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p") pod "network-check-target-7fssr" (UID: "664b3dff-845a-40e3-a987-8ff3d897d228") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:19.640084 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.640050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:19.806398 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.806223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:19.806574 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.806223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:19.806574 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.806471    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:19.806574 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.806236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:19.806574 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.806548    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:19.806791 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:19.806708    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:19.851596 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.851569    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:19.970535 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.970440    2579 generic.go:356] "Generic (PLEG): container finished" podID="4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b" containerID="3ce7f75323ddc635db35ecf1f82b044f3058246f2083753db72120c52eec6961" exitCode=0
Sep 16 18:41:19.970535 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:19.970506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw7zd" event={"ID":"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b","Type":"ContainerDied","Data":"3ce7f75323ddc635db35ecf1f82b044f3058246f2083753db72120c52eec6961"}
Sep 16 18:41:20.280358 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:20.280269    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:20.921848 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:20.921817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:20.975678 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:20.975651    2579 generic.go:356] "Generic (PLEG): container finished" podID="4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b" containerID="b595001e47379885118ea409708d22ec5f70c5e489dfdf9b6340da0415885c46" exitCode=0
Sep 16 18:41:20.975849 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:20.975687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw7zd" event={"ID":"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b","Type":"ContainerDied","Data":"b595001e47379885118ea409708d22ec5f70c5e489dfdf9b6340da0415885c46"}
Sep 16 18:41:21.239681 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:21.239624    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:21.603461 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:21.603428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:21.806222 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:21.806195    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:21.806384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:21.806233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:21.806384 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:21.806206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:21.806384 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:21.806329    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:21.806517 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:21.806383    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:21.806517 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:21.806441    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:21.937892 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:21.937808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:21.981839 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:21.981806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw7zd" event={"ID":"4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b","Type":"ContainerStarted","Data":"9007f25f5ca1bb625852558184b83ce47b1ed5653742fcf337b357cc00f0f468"}
Sep 16 18:41:22.388469 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:22.388441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:22.697785 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:22.697715    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:23.093510 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:23.093482    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:23.263056 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:23.263026    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:23.559381 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:23.559311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:23.805735 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:23.805697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:23.805946 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:23.805706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:23.805946 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:23.805816    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:23.805946 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:23.805871    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:23.805946 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:23.805714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:23.805946 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:23.805933    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:24.051938 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:24.051901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:24.738395 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:24.738360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:25.410190 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:25.410140    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:25.588277 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:25.588244    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:25.807210 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:25.807115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:25.807548 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:25.807222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:25.807548 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:25.807239    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:25.807548 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:25.807292    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:25.807548 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:25.807311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:25.807548 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:25.807393    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:25.993197 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:25.993137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:26.217616 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:26.217588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:26.616936 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:26.616904    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:26.952350 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:26.952276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:27.806544 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:27.806511    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:27.806544 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:27.806533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:27.806742 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:27.806511    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:27.806742 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:27.806624    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:27.806742 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:27.806692    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:27.806838 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:27.806788    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:29.806752 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:29.806712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:29.807257 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:29.806725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:29.807257 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:29.806847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:29.807257 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:29.806841    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:29.807257 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:29.806945    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:29.807257 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:29.807029    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:31.235443 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:31.235406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:31.434270 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:31.434236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:31.806438 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:31.806394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:31.806602 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:31.806447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:31.806602 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:31.806411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:31.806602 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:31.806519    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:31.806697 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:31.806613    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:31.806729 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:31.806708    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:33.774583 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:33.774550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:33.806402 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:33.806373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:33.806556 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:33.806409    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:33.806556 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:33.806372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:33.806556 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:33.806471    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46l2m" podUID="5b52d6a7-170d-47a1-b771-ff55013146d6"
Sep 16 18:41:33.806716 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:33.806568    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7fssr" podUID="664b3dff-845a-40e3-a987-8ff3d897d228"
Sep 16 18:41:33.806716 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:33.806672    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srqqp" podUID="8a2b3c86-57c3-4cd2-8142-5be14426c108"
Sep 16 18:41:34.056054 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.055983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.340572 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.340549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.594928 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.594878    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-67.ec2.internal" event="NodeReady"
Sep 16 18:41:34.595041 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.594964    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:41:34.607251 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.607223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.705638 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.705583    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lw7zd" podStartSLOduration=21.287225598 podStartE2EDuration="48.705563463s" totalImagesPullingTime="27.418337865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:22.027891735 +0000 UTC m=+36.693164184" watchObservedRunningTime="2026-09-16 18:41:34.705563463 +0000 UTC m=+49.370835894"
Sep 16 18:41:34.706175 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.706140    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nl6jb"]
Sep 16 18:41:34.720985 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.720959    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f2h8c"]
Sep 16 18:41:34.721095 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.721079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.723554 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.723535    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9jkmg\""
Sep 16 18:41:34.723672 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.723538    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:41:34.724206 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.724190    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:41:34.737422 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.737404    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nl6jb"]
Sep 16 18:41:34.737503 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.737441    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2h8c"]
Sep 16 18:41:34.737545 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.737514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.740944 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.740928    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:41:34.741021 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.740983    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:41:34.741391 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.741378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d8hzw\""
Sep 16 18:41:34.741519 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.741501    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:41:34.741645 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.741584    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:41:34.804810 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.804627    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zwxfw"]
Sep 16 18:41:34.812733 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.812713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zwxfw"
Sep 16 18:41:34.815803 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.815784    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xc5gc\""
Sep 16 18:41:34.816190 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.816176    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:41:34.816282 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.816263    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:41:34.817017 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.817003    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:41:34.834141 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.834118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zwxfw"]
Sep 16 18:41:34.839764 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.839745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdv4w\" (UniqueName: \"kubernetes.io/projected/0f297a8b-f692-44f8-8b5c-17a8089acfc2-kube-api-access-jdv4w\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.839844 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.839776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5773ddb2-d163-4f2f-aa26-d128fe807649-crio-socket\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.839844 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.839792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm59j\" (UniqueName: \"kubernetes.io/projected/5773ddb2-d163-4f2f-aa26-d128fe807649-kube-api-access-jm59j\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.839844 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.839808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5773ddb2-d163-4f2f-aa26-d128fe807649-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.839942 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.839874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f297a8b-f692-44f8-8b5c-17a8089acfc2-metrics-tls\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.839942 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.839899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5773ddb2-d163-4f2f-aa26-d128fe807649-data-volume\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.839942 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.839935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f297a8b-f692-44f8-8b5c-17a8089acfc2-tmp-dir\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.840032 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.839959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f297a8b-f692-44f8-8b5c-17a8089acfc2-config-volume\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.840032 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.839975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5773ddb2-d163-4f2f-aa26-d128fe807649-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.941111 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm59j\" (UniqueName: \"kubernetes.io/projected/5773ddb2-d163-4f2f-aa26-d128fe807649-kube-api-access-jm59j\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.941111 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5773ddb2-d163-4f2f-aa26-d128fe807649-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.941111 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61f31fe1-86ed-4531-a0a8-783318d9dab8-cert\") pod \"ingress-canary-zwxfw\" (UID: \"61f31fe1-86ed-4531-a0a8-783318d9dab8\") " pod="openshift-ingress-canary/ingress-canary-zwxfw"
Sep 16 18:41:34.941350 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f297a8b-f692-44f8-8b5c-17a8089acfc2-metrics-tls\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.941350 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5773ddb2-d163-4f2f-aa26-d128fe807649-data-volume\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.941430 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f297a8b-f692-44f8-8b5c-17a8089acfc2-tmp-dir\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.941430 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f297a8b-f692-44f8-8b5c-17a8089acfc2-config-volume\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.941509 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5773ddb2-d163-4f2f-aa26-d128fe807649-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.941509 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkdxq\" (UniqueName: \"kubernetes.io/projected/61f31fe1-86ed-4531-a0a8-783318d9dab8-kube-api-access-dkdxq\") pod \"ingress-canary-zwxfw\" (UID: \"61f31fe1-86ed-4531-a0a8-783318d9dab8\") " pod="openshift-ingress-canary/ingress-canary-zwxfw"
Sep 16 18:41:34.941509 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdv4w\" (UniqueName: \"kubernetes.io/projected/0f297a8b-f692-44f8-8b5c-17a8089acfc2-kube-api-access-jdv4w\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.941650 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5773ddb2-d163-4f2f-aa26-d128fe807649-crio-socket\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.941650 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5773ddb2-d163-4f2f-aa26-d128fe807649-data-volume\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.941756 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f297a8b-f692-44f8-8b5c-17a8089acfc2-tmp-dir\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.941756 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5773ddb2-d163-4f2f-aa26-d128fe807649-crio-socket\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.941967 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.941941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f297a8b-f692-44f8-8b5c-17a8089acfc2-config-volume\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.942066 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.942042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5773ddb2-d163-4f2f-aa26-d128fe807649-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.945128 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.945108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5773ddb2-d163-4f2f-aa26-d128fe807649-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.945226 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.945133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f297a8b-f692-44f8-8b5c-17a8089acfc2-metrics-tls\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:34.959638 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.959618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm59j\" (UniqueName: \"kubernetes.io/projected/5773ddb2-d163-4f2f-aa26-d128fe807649-kube-api-access-jm59j\") pod \"insights-runtime-extractor-f2h8c\" (UID: \"5773ddb2-d163-4f2f-aa26-d128fe807649\") " pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:34.959838 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:34.959819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdv4w\" (UniqueName: \"kubernetes.io/projected/0f297a8b-f692-44f8-8b5c-17a8089acfc2-kube-api-access-jdv4w\") pod \"dns-default-nl6jb\" (UID: \"0f297a8b-f692-44f8-8b5c-17a8089acfc2\") " pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:35.030284 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.030261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:35.032306 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.032286    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc"]
Sep 16 18:41:35.042812 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.042786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdxq\" (UniqueName: \"kubernetes.io/projected/61f31fe1-86ed-4531-a0a8-783318d9dab8-kube-api-access-dkdxq\") pod \"ingress-canary-zwxfw\" (UID: \"61f31fe1-86ed-4531-a0a8-783318d9dab8\") " pod="openshift-ingress-canary/ingress-canary-zwxfw"
Sep 16 18:41:35.042897 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.042842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61f31fe1-86ed-4531-a0a8-783318d9dab8-cert\") pod \"ingress-canary-zwxfw\" (UID: \"61f31fe1-86ed-4531-a0a8-783318d9dab8\") " pod="openshift-ingress-canary/ingress-canary-zwxfw"
Sep 16 18:41:35.044046 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.044023    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:35.045006 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.044989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2h8c"
Sep 16 18:41:35.045297 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.045281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61f31fe1-86ed-4531-a0a8-783318d9dab8-cert\") pod \"ingress-canary-zwxfw\" (UID: \"61f31fe1-86ed-4531-a0a8-783318d9dab8\") " pod="openshift-ingress-canary/ingress-canary-zwxfw"
Sep 16 18:41:35.050066 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.050047    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc"
Sep 16 18:41:35.054178 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.054137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:41:35.054321 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.054305    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-trvgp\""
Sep 16 18:41:35.065213 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.065191    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc"]
Sep 16 18:41:35.079526 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.079504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdxq\" (UniqueName: \"kubernetes.io/projected/61f31fe1-86ed-4531-a0a8-783318d9dab8-kube-api-access-dkdxq\") pod \"ingress-canary-zwxfw\" (UID: \"61f31fe1-86ed-4531-a0a8-783318d9dab8\") " pod="openshift-ingress-canary/ingress-canary-zwxfw"
Sep 16 18:41:35.121192 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.121139    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zwxfw"
Sep 16 18:41:35.244331 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.244271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/da54398c-fc8e-43b7-b21c-476d57c695e6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ppgqc\" (UID: \"da54398c-fc8e-43b7-b21c-476d57c695e6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc"
Sep 16 18:41:35.345600 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.345567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/da54398c-fc8e-43b7-b21c-476d57c695e6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ppgqc\" (UID: \"da54398c-fc8e-43b7-b21c-476d57c695e6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc"
Sep 16 18:41:35.347949 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.347931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/da54398c-fc8e-43b7-b21c-476d57c695e6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ppgqc\" (UID: \"da54398c-fc8e-43b7-b21c-476d57c695e6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc"
Sep 16 18:41:35.353318 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.353297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nl6jb"]
Sep 16 18:41:35.356695 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.356670    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2h8c"]
Sep 16 18:41:35.357821 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:35.357802    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f297a8b_f692_44f8_8b5c_17a8089acfc2.slice/crio-ff2a98209b7fc7cc4a1726c75e3f05d71e458d87a1b5ecad31a1931d753b1ef9 WatchSource:0}: Error finding container ff2a98209b7fc7cc4a1726c75e3f05d71e458d87a1b5ecad31a1931d753b1ef9: Status 404 returned error can't find the container with id ff2a98209b7fc7cc4a1726c75e3f05d71e458d87a1b5ecad31a1931d753b1ef9
Sep 16 18:41:35.358614 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.358599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc"
Sep 16 18:41:35.362679 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:35.362635    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5773ddb2_d163_4f2f_aa26_d128fe807649.slice/crio-a22b7cd445118d59bfc08adbcf2a70e0c829b74f89ad162a38dced33d7030cac WatchSource:0}: Error finding container a22b7cd445118d59bfc08adbcf2a70e0c829b74f89ad162a38dced33d7030cac: Status 404 returned error can't find the container with id a22b7cd445118d59bfc08adbcf2a70e0c829b74f89ad162a38dced33d7030cac
Sep 16 18:41:35.435261 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.435214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:35.437868 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.437605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zwxfw"]
Sep 16 18:41:35.438088 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:35.437967    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f31fe1_86ed_4531_a0a8_783318d9dab8.slice/crio-81d1bded1c8cb6f928e2377953261a3e9860febd2920370359abbf9e4341440c WatchSource:0}: Error finding container 81d1bded1c8cb6f928e2377953261a3e9860febd2920370359abbf9e4341440c: Status 404 returned error can't find the container with id 81d1bded1c8cb6f928e2377953261a3e9860febd2920370359abbf9e4341440c
Sep 16 18:41:35.667965 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.667930    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc"]
Sep 16 18:41:35.671331 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:35.671292    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda54398c_fc8e_43b7_b21c_476d57c695e6.slice/crio-e9721ce4a86b78646ab48a3f43777f92e50d0f7566f237d8d638be69a2e9b34c WatchSource:0}: Error finding container e9721ce4a86b78646ab48a3f43777f92e50d0f7566f237d8d638be69a2e9b34c: Status 404 returned error can't find the container with id e9721ce4a86b78646ab48a3f43777f92e50d0f7566f237d8d638be69a2e9b34c
Sep 16 18:41:35.811710 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.811642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:35.812479 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.812451    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:35.812781 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.812753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:35.825330 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.825297    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:35.825741 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.825724    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:35.825925 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.825901    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:35.826144 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.826130    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:35.827247 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.827228    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ndh2n\""
Sep 16 18:41:35.829470 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:35.829450    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5qbcq\""
Sep 16 18:41:36.016745 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:36.016704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2h8c" event={"ID":"5773ddb2-d163-4f2f-aa26-d128fe807649","Type":"ContainerStarted","Data":"9c381e84aafc38f5ff3815ef9465f4ab3c0225c85b9b41fdfe0ebfe43819a6c7"}
Sep 16 18:41:36.016918 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:36.016772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2h8c" event={"ID":"5773ddb2-d163-4f2f-aa26-d128fe807649","Type":"ContainerStarted","Data":"a22b7cd445118d59bfc08adbcf2a70e0c829b74f89ad162a38dced33d7030cac"}
Sep 16 18:41:36.018786 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:36.018750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nl6jb" event={"ID":"0f297a8b-f692-44f8-8b5c-17a8089acfc2","Type":"ContainerStarted","Data":"ff2a98209b7fc7cc4a1726c75e3f05d71e458d87a1b5ecad31a1931d753b1ef9"}
Sep 16 18:41:36.021051 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:36.021022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zwxfw" event={"ID":"61f31fe1-86ed-4531-a0a8-783318d9dab8","Type":"ContainerStarted","Data":"81d1bded1c8cb6f928e2377953261a3e9860febd2920370359abbf9e4341440c"}
Sep 16 18:41:36.023005 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:36.022980    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc" event={"ID":"da54398c-fc8e-43b7-b21c-476d57c695e6","Type":"ContainerStarted","Data":"e9721ce4a86b78646ab48a3f43777f92e50d0f7566f237d8d638be69a2e9b34c"}
Sep 16 18:41:36.719354 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:36.719293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:36.944983 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:36.944907    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:39.033693 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.033659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zwxfw" event={"ID":"61f31fe1-86ed-4531-a0a8-783318d9dab8","Type":"ContainerStarted","Data":"fc627b071a42631856ca59b40c27b6e2fded792361cf794dc3bc0ca51e58af7f"}
Sep 16 18:41:39.035717 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.035690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc" event={"ID":"da54398c-fc8e-43b7-b21c-476d57c695e6","Type":"ContainerStarted","Data":"c95bec73fb6d555a15fb9302bba3f9e8a0d2b38265141399565726f120676b64"}
Sep 16 18:41:39.035916 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.035896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc"
Sep 16 18:41:39.037922 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.037895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2h8c" event={"ID":"5773ddb2-d163-4f2f-aa26-d128fe807649","Type":"ContainerStarted","Data":"f7f42bb7629b0318afd56e016a81bc8e3b26274baca56f2556696c4af5194ae3"}
Sep 16 18:41:39.039993 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.039959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nl6jb" event={"ID":"0f297a8b-f692-44f8-8b5c-17a8089acfc2","Type":"ContainerStarted","Data":"c1c276a6c0dde3bc01311b692621851da7f0fefc2bf53db4fe97c2ff4c80f657"}
Sep 16 18:41:39.040084 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.040008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nl6jb" event={"ID":"0f297a8b-f692-44f8-8b5c-17a8089acfc2","Type":"ContainerStarted","Data":"936e12bc698a3e996df635169c4164e4fd908600a5e6464b3cee2598f1b56e54"}
Sep 16 18:41:39.040145 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.040127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:39.041429 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.041377    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc"
Sep 16 18:41:39.067444 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.067395    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zwxfw" podStartSLOduration=2.412138389 podStartE2EDuration="5.06737691s" totalImagesPullingTime="2.655238521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:39.065387523 +0000 UTC m=+53.730659953" watchObservedRunningTime="2026-09-16 18:41:39.06737691 +0000 UTC m=+53.732649333"
Sep 16 18:41:39.119135 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.119096    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nl6jb" podStartSLOduration=2.388386353 podStartE2EDuration="5.119083921s" totalImagesPullingTime="2.730697568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:39.11828767 +0000 UTC m=+53.783560110" watchObservedRunningTime="2026-09-16 18:41:39.119083921 +0000 UTC m=+53.784356350"
Sep 16 18:41:39.148342 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:39.148293    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppgqc" podStartSLOduration=1.7356529539999999 podStartE2EDuration="4.148274361s" totalImagesPullingTime="2.412621407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:39.148182621 +0000 UTC m=+53.813455072" watchObservedRunningTime="2026-09-16 18:41:39.148274361 +0000 UTC m=+53.813546789"
Sep 16 18:41:40.046303 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.046201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2h8c" event={"ID":"5773ddb2-d163-4f2f-aa26-d128fe807649","Type":"ContainerStarted","Data":"0b5c6e11e882e5e7bcc76df7a6836bd83e65d379b885efd69719782a5160f654"}
Sep 16 18:41:40.066311 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.066259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.118184 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.118123    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f2h8c" podStartSLOduration=1.938307386 podStartE2EDuration="6.118110194s" totalImagesPullingTime="4.179802808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:40.11783176 +0000 UTC m=+54.783104189" watchObservedRunningTime="2026-09-16 18:41:40.118110194 +0000 UTC m=+54.783382631"
Sep 16 18:41:40.208539 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.208517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.254500 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.254473    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"]
Sep 16 18:41:40.256457 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.256443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.261824 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.261801    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:41:40.261924 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.261844    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-khfqc\""
Sep 16 18:41:40.261924 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.261881    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:41:40.261924 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.261801    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:41:40.262199 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.262182    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:41:40.262284 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.262265    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:41:40.278386 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.278364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz696\" (UniqueName: \"kubernetes.io/projected/705435ab-3aa7-4453-b731-81b340275263-kube-api-access-rz696\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.278469 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.278413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/705435ab-3aa7-4453-b731-81b340275263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.278521 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.278480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/705435ab-3aa7-4453-b731-81b340275263-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.278521 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.278509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/705435ab-3aa7-4453-b731-81b340275263-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.312024 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.311971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"]
Sep 16 18:41:40.336114 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.336091    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.379366 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.379347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/705435ab-3aa7-4453-b731-81b340275263-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.379435 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.379372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/705435ab-3aa7-4453-b731-81b340275263-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.379528 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.379507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz696\" (UniqueName: \"kubernetes.io/projected/705435ab-3aa7-4453-b731-81b340275263-kube-api-access-rz696\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.379571 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.379558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/705435ab-3aa7-4453-b731-81b340275263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.380201 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.380111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/705435ab-3aa7-4453-b731-81b340275263-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.381844 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.381827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/705435ab-3aa7-4453-b731-81b340275263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.381919 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.381881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/705435ab-3aa7-4453-b731-81b340275263-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.389854 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.389832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz696\" (UniqueName: \"kubernetes.io/projected/705435ab-3aa7-4453-b731-81b340275263-kube-api-access-rz696\") pod \"prometheus-operator-68bcd7f987-bfm22\" (UID: \"705435ab-3aa7-4453-b731-81b340275263\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.564903 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.564840    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"
Sep 16 18:41:40.842288 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:40.842260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-bfm22"]
Sep 16 18:41:40.845643 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:40.845611    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705435ab_3aa7_4453_b731_81b340275263.slice/crio-4f7609b26eec0d7fd5d6c008f7e939addf403a2a0a284ca33663bd9dd2ca3af5 WatchSource:0}: Error finding container 4f7609b26eec0d7fd5d6c008f7e939addf403a2a0a284ca33663bd9dd2ca3af5: Status 404 returned error can't find the container with id 4f7609b26eec0d7fd5d6c008f7e939addf403a2a0a284ca33663bd9dd2ca3af5
Sep 16 18:41:41.049950 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:41.049918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22" event={"ID":"705435ab-3aa7-4453-b731-81b340275263","Type":"ContainerStarted","Data":"4f7609b26eec0d7fd5d6c008f7e939addf403a2a0a284ca33663bd9dd2ca3af5"}
Sep 16 18:41:41.082359 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:41.082330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.301847 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:41.301772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:42.066770 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.066734    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f7f88989c-9rr5p"]
Sep 16 18:41:42.073684 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.073657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.078305 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.078283    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:41:42.079334 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.079280    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:41:42.079451 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.079384    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:41:42.079520 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.079280    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:41:42.081637 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.081430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:41:42.081637 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.081447    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:41:42.081637 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.081520    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:41:42.081637 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.081621    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c54mj\""
Sep 16 18:41:42.084810 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.084787    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:41:42.089410 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.089387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-oauth-serving-cert\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.089523 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.089452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-serving-cert\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.089626 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.089568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-trusted-ca-bundle\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.089626 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.089601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-config\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.089801 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.089626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cc6v\" (UniqueName: \"kubernetes.io/projected/a6f88f9d-22aa-4985-b319-bfdef20ed5de-kube-api-access-8cc6v\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.089801 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.089654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-service-ca\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.089801 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.089683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-oauth-config\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.091307 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.091284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7f88989c-9rr5p"]
Sep 16 18:41:42.190402 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.190366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-oauth-serving-cert\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.190585 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.190420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-serving-cert\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.190585 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.190450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-trusted-ca-bundle\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.190585 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.190465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-config\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.190585 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.190479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cc6v\" (UniqueName: \"kubernetes.io/projected/a6f88f9d-22aa-4985-b319-bfdef20ed5de-kube-api-access-8cc6v\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.190585 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.190496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-service-ca\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.190850 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.190638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-oauth-config\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.191130 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.191088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-oauth-serving-cert\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.191457 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.191435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-service-ca\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.191699 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.191673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-config\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.191893 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.191873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-trusted-ca-bundle\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.193238 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.193215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-oauth-config\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.193308 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.193270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-serving-cert\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.201934 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.201911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cc6v\" (UniqueName: \"kubernetes.io/projected/a6f88f9d-22aa-4985-b319-bfdef20ed5de-kube-api-access-8cc6v\") pod \"console-7f7f88989c-9rr5p\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") " pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.385571 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.385541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:42.603446 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.603418    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:42.762051 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.762010    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:42.808751 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:42.808720    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7f88989c-9rr5p"]
Sep 16 18:41:42.811796 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:42.811755    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f88f9d_22aa_4985_b319_bfdef20ed5de.slice/crio-f1bde6bbd66f5a28fbc9301d3efd45c0a57f810e68de7938480ef691a3e9f540 WatchSource:0}: Error finding container f1bde6bbd66f5a28fbc9301d3efd45c0a57f810e68de7938480ef691a3e9f540: Status 404 returned error can't find the container with id f1bde6bbd66f5a28fbc9301d3efd45c0a57f810e68de7938480ef691a3e9f540
Sep 16 18:41:43.015894 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:43.015822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.057020 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:43.056986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f88989c-9rr5p" event={"ID":"a6f88f9d-22aa-4985-b319-bfdef20ed5de","Type":"ContainerStarted","Data":"f1bde6bbd66f5a28fbc9301d3efd45c0a57f810e68de7938480ef691a3e9f540"}
Sep 16 18:41:43.058851 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:43.058827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22" event={"ID":"705435ab-3aa7-4453-b731-81b340275263","Type":"ContainerStarted","Data":"2f89c66de21dffe5100bfdc38c7a32f2802a04fe60319cf2819fc20494aa27a3"}
Sep 16 18:41:43.058966 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:43.058870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22" event={"ID":"705435ab-3aa7-4453-b731-81b340275263","Type":"ContainerStarted","Data":"c49b02d68ffa048125e4df2ea474a91626780222d818ec5cf0ffec92b27d6c50"}
Sep 16 18:41:43.083025 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:43.082967    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfm22" podStartSLOduration=1.405278105 podStartE2EDuration="3.082952197s" totalImagesPullingTime="1.677674092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:43.082491629 +0000 UTC m=+57.747764080" watchObservedRunningTime="2026-09-16 18:41:43.082952197 +0000 UTC m=+57.748224627"
Sep 16 18:41:43.278938 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:43.278863    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.397920 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:43.397888    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.460851 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:44.460806    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.763528 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:44.763396    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.971580 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:44.971518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.066116 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.066049    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"]
Sep 16 18:41:45.072106 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.072079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.079879 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.079741    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-x26r5\""
Sep 16 18:41:45.079879 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.079801    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:41:45.080753 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.080727    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:41:45.091657 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.091629    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"]
Sep 16 18:41:45.113445 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.113177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cfc7333-89d1-44f5-9d85-27547bcd574c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.113445 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.113236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cfc7333-89d1-44f5-9d85-27547bcd574c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.113445 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.113274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4cfc7333-89d1-44f5-9d85-27547bcd574c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.113445 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.113380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6vp\" (UniqueName: \"kubernetes.io/projected/4cfc7333-89d1-44f5-9d85-27547bcd574c-kube-api-access-fr6vp\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.117045 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.117020    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"]
Sep 16 18:41:45.120925 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.120900    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.121405 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.121381    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hwz99"]
Sep 16 18:41:45.123774 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.123750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.126540 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.126249    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:41:45.126540 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.126333    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rs5pn\""
Sep 16 18:41:45.126540 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.126379    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nr5ld\""
Sep 16 18:41:45.126540 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.126401    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:41:45.126540 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.126438    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:41:45.126837 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.126566    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:41:45.126837 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.126681    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:41:45.126837 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.126766    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:41:45.154168 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.153725    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"]
Sep 16 18:41:45.213937 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.213905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4cfc7333-89d1-44f5-9d85-27547bcd574c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.214067 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.213954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-wtmp\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.214119 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-textfile\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.214212 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6ckm\" (UniqueName: \"kubernetes.io/projected/03e62226-e882-4e47-8f9f-d98e180d3205-kube-api-access-m6ckm\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.214212 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03e62226-e882-4e47-8f9f-d98e180d3205-sys\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.214316 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.214375 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssj22\" (UniqueName: \"kubernetes.io/projected/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-api-access-ssj22\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.214375 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03e62226-e882-4e47-8f9f-d98e180d3205-metrics-client-ca\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.214464 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cfc7333-89d1-44f5-9d85-27547bcd574c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.214464 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.214464 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6vp\" (UniqueName: \"kubernetes.io/projected/4cfc7333-89d1-44f5-9d85-27547bcd574c-kube-api-access-fr6vp\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.214576 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.214576 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.214576 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-tls\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.214576 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.214711 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.214711 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03e62226-e882-4e47-8f9f-d98e180d3205-root\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.214711 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-accelerators-collector-config\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.214837 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.214734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cfc7333-89d1-44f5-9d85-27547bcd574c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.215464 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.215441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cfc7333-89d1-44f5-9d85-27547bcd574c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.216663 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.216641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4cfc7333-89d1-44f5-9d85-27547bcd574c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.223289 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.223244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cfc7333-89d1-44f5-9d85-27547bcd574c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.232843 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.232818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6vp\" (UniqueName: \"kubernetes.io/projected/4cfc7333-89d1-44f5-9d85-27547bcd574c-kube-api-access-fr6vp\") pod \"openshift-state-metrics-67dcf8c979-mm67w\" (UID: \"4cfc7333-89d1-44f5-9d85-27547bcd574c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.290949 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.290918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.315095 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-accelerators-collector-config\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315232 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-wtmp\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315232 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-textfile\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315232 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6ckm\" (UniqueName: \"kubernetes.io/projected/03e62226-e882-4e47-8f9f-d98e180d3205-kube-api-access-m6ckm\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315232 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03e62226-e882-4e47-8f9f-d98e180d3205-sys\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315428 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.315428 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssj22\" (UniqueName: \"kubernetes.io/projected/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-api-access-ssj22\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.315428 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315289    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-wtmp\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315428 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03e62226-e882-4e47-8f9f-d98e180d3205-metrics-client-ca\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315428 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.315428 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03e62226-e882-4e47-8f9f-d98e180d3205-sys\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315428 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315428 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.315428 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-tls\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315836 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.315836 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.315836 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03e62226-e882-4e47-8f9f-d98e180d3205-root\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315836 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-textfile\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315836 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315588    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03e62226-e882-4e47-8f9f-d98e180d3205-root\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.315836 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-accelerators-collector-config\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.316066 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.315871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.316787 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.316190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03e62226-e882-4e47-8f9f-d98e180d3205-metrics-client-ca\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.316787 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.316253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.316787 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:45.316286    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:41:45.316787 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:41:45.316501    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-tls podName:03e62226-e882-4e47-8f9f-d98e180d3205 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.816482217 +0000 UTC m=+60.481754632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-tls") pod "node-exporter-hwz99" (UID: "03e62226-e882-4e47-8f9f-d98e180d3205") : secret "node-exporter-tls" not found
Sep 16 18:41:45.316787 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.316716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.318352 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.318333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.318885 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.318860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.319499 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.319480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.336833 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.336806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6ckm\" (UniqueName: \"kubernetes.io/projected/03e62226-e882-4e47-8f9f-d98e180d3205-kube-api-access-m6ckm\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.339217 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.339195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssj22\" (UniqueName: \"kubernetes.io/projected/c5aba59c-7e57-4c03-9449-abd9dd2e40ba-kube-api-access-ssj22\") pod \"kube-state-metrics-59fbdd77d-bj85p\" (UID: \"c5aba59c-7e57-4c03-9449-abd9dd2e40ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.383345 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.383318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"
Sep 16 18:41:45.432314 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.432007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"
Sep 16 18:41:45.496839 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.496807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.648639 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.648612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.670899 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.670870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w"]
Sep 16 18:41:45.673362 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:45.673336    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfc7333_89d1_44f5_9d85_27547bcd574c.slice/crio-650b8f67398c4e4b378eaf95a5028e9210ccdf57283fd4f27a8f76ef6b9e3198 WatchSource:0}: Error finding container 650b8f67398c4e4b378eaf95a5028e9210ccdf57283fd4f27a8f76ef6b9e3198: Status 404 returned error can't find the container with id 650b8f67398c4e4b378eaf95a5028e9210ccdf57283fd4f27a8f76ef6b9e3198
Sep 16 18:41:45.723871 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.723844    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p"]
Sep 16 18:41:45.726434 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:45.726413    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5aba59c_7e57_4c03_9449_abd9dd2e40ba.slice/crio-6a2f1bd964ac3234bddc5028476c6037c997f365dc75464329242a208224763b WatchSource:0}: Error finding container 6a2f1bd964ac3234bddc5028476c6037c997f365dc75464329242a208224763b: Status 404 returned error can't find the container with id 6a2f1bd964ac3234bddc5028476c6037c997f365dc75464329242a208224763b
Sep 16 18:41:45.818166 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.818120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-tls\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.821127 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.821100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03e62226-e882-4e47-8f9f-d98e180d3205-node-exporter-tls\") pod \"node-exporter-hwz99\" (UID: \"03e62226-e882-4e47-8f9f-d98e180d3205\") " pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:45.864372 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.864343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.969803 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:45.969733    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smblz"
Sep 16 18:41:46.056466 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.056432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.065362 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.065334    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nr5ld\""
Sep 16 18:41:46.071659 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.071632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f88989c-9rr5p" event={"ID":"a6f88f9d-22aa-4985-b319-bfdef20ed5de","Type":"ContainerStarted","Data":"348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2"}
Sep 16 18:41:46.072563 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.072543    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hwz99"
Sep 16 18:41:46.072892 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.072872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p" event={"ID":"c5aba59c-7e57-4c03-9449-abd9dd2e40ba","Type":"ContainerStarted","Data":"6a2f1bd964ac3234bddc5028476c6037c997f365dc75464329242a208224763b"}
Sep 16 18:41:46.074766 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.074745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w" event={"ID":"4cfc7333-89d1-44f5-9d85-27547bcd574c","Type":"ContainerStarted","Data":"13792d75b6535108f2fa321dbef13e4fb88a071c985d14106c8f22682bbb9887"}
Sep 16 18:41:46.074844 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.074799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w" event={"ID":"4cfc7333-89d1-44f5-9d85-27547bcd574c","Type":"ContainerStarted","Data":"41f0556d55dff7dd125c790890918a200ed07d63b540cd4d59573f680a339dac"}
Sep 16 18:41:46.074844 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.074828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w" event={"ID":"4cfc7333-89d1-44f5-9d85-27547bcd574c","Type":"ContainerStarted","Data":"650b8f67398c4e4b378eaf95a5028e9210ccdf57283fd4f27a8f76ef6b9e3198"}
Sep 16 18:41:46.082421 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:46.082400    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e62226_e882_4e47_8f9f_d98e180d3205.slice/crio-48704e55995d01fbb17bd8c3993f744996f57977b8e0cd25bbc7a0576fe9c78f WatchSource:0}: Error finding container 48704e55995d01fbb17bd8c3993f744996f57977b8e0cd25bbc7a0576fe9c78f: Status 404 returned error can't find the container with id 48704e55995d01fbb17bd8c3993f744996f57977b8e0cd25bbc7a0576fe9c78f
Sep 16 18:41:46.108860 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.108813    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f7f88989c-9rr5p" podStartSLOduration=1.503004821 podStartE2EDuration="4.108797333s" totalImagesPullingTime="2.605792512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:46.108417786 +0000 UTC m=+60.773690217" watchObservedRunningTime="2026-09-16 18:41:46.108797333 +0000 UTC m=+60.774069763"
Sep 16 18:41:46.141912 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.141882    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:41:46.144791 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.144775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.147537 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.147505    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:41:46.147537 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.147535    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:41:46.147697 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.147545    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:41:46.147697 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.147535    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:41:46.147697 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.147534    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:41:46.147976 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.147959    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:41:46.148060 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.147958    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d9bg9\""
Sep 16 18:41:46.148060 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.147988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:41:46.148186 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.148177    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:41:46.148398 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.148383    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:41:46.181525 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.181494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:41:46.220955 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.220878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5cafeaa-779d-4c84-a719-8af92f10e364-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.220955 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.220939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzmmn\" (UniqueName: \"kubernetes.io/projected/b5cafeaa-779d-4c84-a719-8af92f10e364-kube-api-access-mzmmn\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221264 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-web-config\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221264 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5cafeaa-779d-4c84-a719-8af92f10e364-config-out\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221264 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221264 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221264 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221434 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5cafeaa-779d-4c84-a719-8af92f10e364-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221434 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221434 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221434 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5cafeaa-779d-4c84-a719-8af92f10e364-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221434 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5cafeaa-779d-4c84-a719-8af92f10e364-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.221576 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.221459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322753 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5cafeaa-779d-4c84-a719-8af92f10e364-config-out\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322753 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5cafeaa-779d-4c84-a719-8af92f10e364-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5cafeaa-779d-4c84-a719-8af92f10e364-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5cafeaa-779d-4c84-a719-8af92f10e364-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.322984 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.322973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5cafeaa-779d-4c84-a719-8af92f10e364-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.323355 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.323011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmmn\" (UniqueName: \"kubernetes.io/projected/b5cafeaa-779d-4c84-a719-8af92f10e364-kube-api-access-mzmmn\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.323355 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.323038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-web-config\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.325650 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.323951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5cafeaa-779d-4c84-a719-8af92f10e364-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.325650 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.324571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5cafeaa-779d-4c84-a719-8af92f10e364-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.325650 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.324900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5cafeaa-779d-4c84-a719-8af92f10e364-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.329031 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.326735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.329031 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.326841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.329031 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.327516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.329031 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.327662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.329031 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.327782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-web-config\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.329031 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.328121    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.329397 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.329376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5cafeaa-779d-4c84-a719-8af92f10e364-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.330668 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.330225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5cafeaa-779d-4c84-a719-8af92f10e364-config-out\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.330668 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.330628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5cafeaa-779d-4c84-a719-8af92f10e364-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.336133 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.336109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzmmn\" (UniqueName: \"kubernetes.io/projected/b5cafeaa-779d-4c84-a719-8af92f10e364-kube-api-access-mzmmn\") pod \"alertmanager-main-0\" (UID: \"b5cafeaa-779d-4c84-a719-8af92f10e364\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.360166 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.360123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.454249 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.454201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:46.552019 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.551987    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.768634 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.768555    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:41:46.892104 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:46.892058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.073133 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:47.073106    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cafeaa_779d_4c84_a719_8af92f10e364.slice/crio-b9209a44bc774f50dcbc10fdb6d2fe792f7b66749a63fa5a3996b2b30949620a WatchSource:0}: Error finding container b9209a44bc774f50dcbc10fdb6d2fe792f7b66749a63fa5a3996b2b30949620a: Status 404 returned error can't find the container with id b9209a44bc774f50dcbc10fdb6d2fe792f7b66749a63fa5a3996b2b30949620a
Sep 16 18:41:47.079101 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:47.079075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5cafeaa-779d-4c84-a719-8af92f10e364","Type":"ContainerStarted","Data":"b9209a44bc774f50dcbc10fdb6d2fe792f7b66749a63fa5a3996b2b30949620a"}
Sep 16 18:41:47.080356 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:47.080332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hwz99" event={"ID":"03e62226-e882-4e47-8f9f-d98e180d3205","Type":"ContainerStarted","Data":"48704e55995d01fbb17bd8c3993f744996f57977b8e0cd25bbc7a0576fe9c78f"}
Sep 16 18:41:47.999000 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:47.998962    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:48.038046 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.038006    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9b49dff5d-g777h"]
Sep 16 18:41:48.041243 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.041221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.044275 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.044252    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:41:48.044418 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.044400    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:41:48.044634 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.044610    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:41:48.044764 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.044639    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:41:48.044962 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.044945    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5tdpa0f3cojcg\""
Sep 16 18:41:48.045043 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.044947    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:41:48.045043 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.044997    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q9v2s\""
Sep 16 18:41:48.062386 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.062361    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9b49dff5d-g777h"]
Sep 16 18:41:48.088507 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.088472    2579 generic.go:356] "Generic (PLEG): container finished" podID="03e62226-e882-4e47-8f9f-d98e180d3205" containerID="2c25a275e57a2354919e6fa74fee008b06147b481388b9e0bdec7614b7e0196b" exitCode=0
Sep 16 18:41:48.088622 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.088558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hwz99" event={"ID":"03e62226-e882-4e47-8f9f-d98e180d3205","Type":"ContainerDied","Data":"2c25a275e57a2354919e6fa74fee008b06147b481388b9e0bdec7614b7e0196b"}
Sep 16 18:41:48.090855 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.090834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p" event={"ID":"c5aba59c-7e57-4c03-9449-abd9dd2e40ba","Type":"ContainerStarted","Data":"b49cc5d6bae5b6e6d9edb9ffc99e4a5d717122383e90e5200e4dc28c6a461a2c"}
Sep 16 18:41:48.090933 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.090892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p" event={"ID":"c5aba59c-7e57-4c03-9449-abd9dd2e40ba","Type":"ContainerStarted","Data":"0d886f7326120c1faa8697d143774fa1f3f30a26ffa3ff93cf21e983072a4dd8"}
Sep 16 18:41:48.090933 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.090913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p" event={"ID":"c5aba59c-7e57-4c03-9449-abd9dd2e40ba","Type":"ContainerStarted","Data":"e4ffd00fe7999978bf81020960ef276c1deabcebab3c9b60267ec18a4a011dd7"}
Sep 16 18:41:48.092990 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.092970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w" event={"ID":"4cfc7333-89d1-44f5-9d85-27547bcd574c","Type":"ContainerStarted","Data":"058a83d4ba0d065e0fe2ca15fb38cb8c667bb3007d5ff2cf23e3970ede8620ae"}
Sep 16 18:41:48.138701 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.138197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.138701 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.138236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.138701 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.138263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-metrics-client-ca\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.138701 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.138320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.138701 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.138453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.138701 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.138524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-grpc-tls\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.139092 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.138843    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzt5t\" (UniqueName: \"kubernetes.io/projected/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-kube-api-access-qzt5t\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.139092 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.138877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-tls\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.168596 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.168553    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bj85p" podStartSLOduration=1.5900084479999999 podStartE2EDuration="3.168537528s" totalImagesPullingTime="1.57852908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:48.165835435 +0000 UTC m=+62.831107876" watchObservedRunningTime="2026-09-16 18:41:48.168537528 +0000 UTC m=+62.833809958"
Sep 16 18:41:48.191888 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.191863    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:48.240292 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.240268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.240397 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.240311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-grpc-tls\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.240516 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.240497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzt5t\" (UniqueName: \"kubernetes.io/projected/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-kube-api-access-qzt5t\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.240552 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.240533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-tls\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.240614 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.240599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.240651 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.240630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.240705 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.240656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-metrics-client-ca\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.240705 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.240686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.241710 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.241680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-metrics-client-ca\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.243198 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.243117    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mm67w" podStartSLOduration=1.725444531 podStartE2EDuration="3.243103632s" totalImagesPullingTime="1.517659101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:48.241882205 +0000 UTC m=+62.907154653" watchObservedRunningTime="2026-09-16 18:41:48.243103632 +0000 UTC m=+62.908376052"
Sep 16 18:41:48.243632 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.243604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.243728 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.243693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.243890 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.243869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-grpc-tls\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.244061 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.244042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.244261 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.244239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-tls\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.245244 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.245224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.250542 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.250491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzt5t\" (UniqueName: \"kubernetes.io/projected/04aee5ca-0cf7-489a-b9e8-09592e9c12d7-kube-api-access-qzt5t\") pod \"thanos-querier-9b49dff5d-g777h\" (UID: \"04aee5ca-0cf7-489a-b9e8-09592e9c12d7\") " pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.352118 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.352084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:48.582370 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.582336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:48.642186 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.641926    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9b49dff5d-g777h"]
Sep 16 18:41:48.644601 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:48.644574    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04aee5ca_0cf7_489a_b9e8_09592e9c12d7.slice/crio-0d6c42df5299cfeec83045f68e10c3a1089f59e94b426bea7448e9f57a32a975 WatchSource:0}: Error finding container 0d6c42df5299cfeec83045f68e10c3a1089f59e94b426bea7448e9f57a32a975: Status 404 returned error can't find the container with id 0d6c42df5299cfeec83045f68e10c3a1089f59e94b426bea7448e9f57a32a975
Sep 16 18:41:48.735245 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.735213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:48.900535 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:48.900448    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.049318 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.049290    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nl6jb"
Sep 16 18:41:49.098880 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.098849    2579 generic.go:356] "Generic (PLEG): container finished" podID="b5cafeaa-779d-4c84-a719-8af92f10e364" containerID="dc9b5f69bf9930ca30c3a6a98e4184367d9f353933c2a2a9355eea405e4a6125" exitCode=0
Sep 16 18:41:49.099070 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.098933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5cafeaa-779d-4c84-a719-8af92f10e364","Type":"ContainerDied","Data":"dc9b5f69bf9930ca30c3a6a98e4184367d9f353933c2a2a9355eea405e4a6125"}
Sep 16 18:41:49.100328 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.100303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h" event={"ID":"04aee5ca-0cf7-489a-b9e8-09592e9c12d7","Type":"ContainerStarted","Data":"0d6c42df5299cfeec83045f68e10c3a1089f59e94b426bea7448e9f57a32a975"}
Sep 16 18:41:49.102826 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.102804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hwz99" event={"ID":"03e62226-e882-4e47-8f9f-d98e180d3205","Type":"ContainerStarted","Data":"df84d8a9d9831face52efdc94a93abc82e3331224a725a5a1270cf2a523b6c2d"}
Sep 16 18:41:49.102912 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.102847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hwz99" event={"ID":"03e62226-e882-4e47-8f9f-d98e180d3205","Type":"ContainerStarted","Data":"5f68ec3d2520f3ec7d89621b53de490263532860d5a49811621711eb120b4593"}
Sep 16 18:41:49.179234 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.179134    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.265315 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.265254    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hwz99" podStartSLOduration=2.942990228 podStartE2EDuration="4.265235492s" totalImagesPullingTime="1.226757188s" totalInitContainerRuntime="95.488076ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:49.26500708 +0000 UTC m=+63.930279530" watchObservedRunningTime="2026-09-16 18:41:49.265235492 +0000 UTC m=+63.930507933"
Sep 16 18:41:49.551722 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.551648    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"]
Sep 16 18:41:49.568029 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.567998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"
Sep 16 18:41:49.571445 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.571420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:41:49.571656 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.571640    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nk6lc\""
Sep 16 18:41:49.575695 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.575670    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"]
Sep 16 18:41:49.581522 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.581497    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.654493 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.654459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8989cb34-2530-415a-bb8a-1a3542e6cfff-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-59pp5\" (UID: \"8989cb34-2530-415a-bb8a-1a3542e6cfff\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"
Sep 16 18:41:49.654668 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.654550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbll\" (UniqueName: \"kubernetes.io/projected/8989cb34-2530-415a-bb8a-1a3542e6cfff-kube-api-access-trbll\") pod \"monitoring-plugin-5d85778b9d-59pp5\" (UID: \"8989cb34-2530-415a-bb8a-1a3542e6cfff\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"
Sep 16 18:41:49.736627 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.736591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.755540 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.755509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8989cb34-2530-415a-bb8a-1a3542e6cfff-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-59pp5\" (UID: \"8989cb34-2530-415a-bb8a-1a3542e6cfff\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"
Sep 16 18:41:49.755704 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.755582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbll\" (UniqueName: \"kubernetes.io/projected/8989cb34-2530-415a-bb8a-1a3542e6cfff-kube-api-access-trbll\") pod \"monitoring-plugin-5d85778b9d-59pp5\" (UID: \"8989cb34-2530-415a-bb8a-1a3542e6cfff\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"
Sep 16 18:41:49.758827 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.758805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8989cb34-2530-415a-bb8a-1a3542e6cfff-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-59pp5\" (UID: \"8989cb34-2530-415a-bb8a-1a3542e6cfff\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"
Sep 16 18:41:49.767834 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.767803    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64644594b-5c8cb"]
Sep 16 18:41:49.780723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.780700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbll\" (UniqueName: \"kubernetes.io/projected/8989cb34-2530-415a-bb8a-1a3542e6cfff-kube-api-access-trbll\") pod \"monitoring-plugin-5d85778b9d-59pp5\" (UID: \"8989cb34-2530-415a-bb8a-1a3542e6cfff\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"
Sep 16 18:41:49.790856 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.790832    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.795657 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.795553    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:41:49.797369 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.797348    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lr7kf\""
Sep 16 18:41:49.797653 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.797354    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:41:49.798956 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.798525    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:41:49.799074 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.799031    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ac2r87lar03n1\""
Sep 16 18:41:49.799492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.799473    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:41:49.803456 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.800630    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64644594b-5c8cb"]
Sep 16 18:41:49.854076 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.854045    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.856437 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.856406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b651e8e0-92d6-40b2-8064-7523c2ca41a8-secret-metrics-server-tls\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.856572 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.856480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b651e8e0-92d6-40b2-8064-7523c2ca41a8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.856629 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.856581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b651e8e0-92d6-40b2-8064-7523c2ca41a8-secret-metrics-server-client-certs\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.856689 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.856662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b651e8e0-92d6-40b2-8064-7523c2ca41a8-audit-log\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.856739 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.856691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b651e8e0-92d6-40b2-8064-7523c2ca41a8-metrics-server-audit-profiles\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.856739 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.856722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7qqd\" (UniqueName: \"kubernetes.io/projected/b651e8e0-92d6-40b2-8064-7523c2ca41a8-kube-api-access-v7qqd\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.856836 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.856793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b651e8e0-92d6-40b2-8064-7523c2ca41a8-client-ca-bundle\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.880698 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.880317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"
Sep 16 18:41:49.957452 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.957411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b651e8e0-92d6-40b2-8064-7523c2ca41a8-audit-log\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.957569 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.957461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b651e8e0-92d6-40b2-8064-7523c2ca41a8-metrics-server-audit-profiles\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.957569 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.957491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7qqd\" (UniqueName: \"kubernetes.io/projected/b651e8e0-92d6-40b2-8064-7523c2ca41a8-kube-api-access-v7qqd\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.957569 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.957543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b651e8e0-92d6-40b2-8064-7523c2ca41a8-client-ca-bundle\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.957758 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.957579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b651e8e0-92d6-40b2-8064-7523c2ca41a8-secret-metrics-server-tls\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.957758 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.957636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b651e8e0-92d6-40b2-8064-7523c2ca41a8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.957758 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.957693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b651e8e0-92d6-40b2-8064-7523c2ca41a8-secret-metrics-server-client-certs\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.958495 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.958447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b651e8e0-92d6-40b2-8064-7523c2ca41a8-audit-log\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.960586 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.960556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b651e8e0-92d6-40b2-8064-7523c2ca41a8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.960927 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.960881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b651e8e0-92d6-40b2-8064-7523c2ca41a8-metrics-server-audit-profiles\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.963872 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.963787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b651e8e0-92d6-40b2-8064-7523c2ca41a8-secret-metrics-server-tls\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.965472 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.965398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b651e8e0-92d6-40b2-8064-7523c2ca41a8-client-ca-bundle\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.966737 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.966712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b651e8e0-92d6-40b2-8064-7523c2ca41a8-secret-metrics-server-client-certs\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:49.978801 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:49.978773    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7qqd\" (UniqueName: \"kubernetes.io/projected/b651e8e0-92d6-40b2-8064-7523c2ca41a8-kube-api-access-v7qqd\") pod \"metrics-server-64644594b-5c8cb\" (UID: \"b651e8e0-92d6-40b2-8064-7523c2ca41a8\") " pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:50.107679 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.107638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:41:50.191761 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.191356    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"]
Sep 16 18:41:50.329034 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.329002    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75ff4865fc-9mq57"]
Sep 16 18:41:50.343090 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.343061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.346026 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.345993    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:41:50.346426 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.346189    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:41:50.346426 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.346189    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:41:50.346426 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.346291    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kw9xc\""
Sep 16 18:41:50.346426 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.346325    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:41:50.346426 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.346394    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:41:50.352896 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.352873    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:41:50.355660 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.355635    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75ff4865fc-9mq57"]
Sep 16 18:41:50.361286 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.361233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-federate-client-tls\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.361286 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.361278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-telemeter-client-tls\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.361432 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.361365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.361432 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.361422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-metrics-client-ca\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.361517 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.361455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n5gb\" (UniqueName: \"kubernetes.io/projected/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-kube-api-access-9n5gb\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.361572 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.361528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-secret-telemeter-client\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.361572 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.361558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.361670 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.361586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-serving-certs-ca-bundle\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.430205 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.430171    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64644594b-5c8cb"]
Sep 16 18:41:50.465438 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.465404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-metrics-client-ca\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.465453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n5gb\" (UniqueName: \"kubernetes.io/projected/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-kube-api-access-9n5gb\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.465618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-secret-telemeter-client\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.465644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.465675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-serving-certs-ca-bundle\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.465768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-federate-client-tls\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.465817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-telemeter-client-tls\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.466409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.467127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.467368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-serving-certs-ca-bundle\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.469389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-secret-telemeter-client\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.469847    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-federate-client-tls\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.470237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.471441 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.471213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-telemeter-client-tls\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.472783 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.472756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-metrics-client-ca\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.481090 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.481058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n5gb\" (UniqueName: \"kubernetes.io/projected/4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154-kube-api-access-9n5gb\") pod \"telemeter-client-75ff4865fc-9mq57\" (UID: \"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154\") " pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.503201 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.503138    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7f88989c-9rr5p"]
Sep 16 18:41:50.510765 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.510713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:50.543648 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.543618    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bf654d699-cbg4l"]
Sep 16 18:41:50.569237 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.569194    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bf654d699-cbg4l"]
Sep 16 18:41:50.569432 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.569328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.616102 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.616024    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:50.657263 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.657227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57"
Sep 16 18:41:50.668085 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.668045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-trusted-ca-bundle\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.668279 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.668113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-oauth-config\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.668279 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.668225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcrj4\" (UniqueName: \"kubernetes.io/projected/bc7d98d4-c658-4469-8908-5f8610fc8d9f-kube-api-access-tcrj4\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.668363 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.668286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-oauth-serving-cert\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.668363 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.668319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-service-ca\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.668363 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.668347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-serving-cert\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.668454 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.668368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-config\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.768975 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.768764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-trusted-ca-bundle\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.768975 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.768846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-oauth-config\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.768975 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.768881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrj4\" (UniqueName: \"kubernetes.io/projected/bc7d98d4-c658-4469-8908-5f8610fc8d9f-kube-api-access-tcrj4\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.768975 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.768920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-oauth-serving-cert\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.768975 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.768977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-service-ca\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.769483 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.769018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-serving-cert\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.769483 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.769056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-config\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.770196 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.769782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-trusted-ca-bundle\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.770196 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.769851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-oauth-serving-cert\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.770196 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.769955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-service-ca\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.770587 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.770286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-config\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.772288 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.772265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-oauth-config\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.772288 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.772281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-serving-cert\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.784477 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.784453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcrj4\" (UniqueName: \"kubernetes.io/projected/bc7d98d4-c658-4469-8908-5f8610fc8d9f-kube-api-access-tcrj4\") pod \"console-bf654d699-cbg4l\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") " pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:50.839687 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:50.839646    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8989cb34_2530_415a_bb8a_1a3542e6cfff.slice/crio-0594f1f501ddb6fed144401aeb2450068db28b6f7a1ac9aa0f8732992fe3c3eb WatchSource:0}: Error finding container 0594f1f501ddb6fed144401aeb2450068db28b6f7a1ac9aa0f8732992fe3c3eb: Status 404 returned error can't find the container with id 0594f1f501ddb6fed144401aeb2450068db28b6f7a1ac9aa0f8732992fe3c3eb
Sep 16 18:41:50.880414 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:50.880329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:41:51.114775 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.114735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5" event={"ID":"8989cb34-2530-415a-bb8a-1a3542e6cfff","Type":"ContainerStarted","Data":"0594f1f501ddb6fed144401aeb2450068db28b6f7a1ac9aa0f8732992fe3c3eb"}
Sep 16 18:41:51.130579 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.130446    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:51.211522 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:51.211481    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb651e8e0_92d6_40b2_8064_7523c2ca41a8.slice/crio-308356763d43d634bc51b068aec29ba7b814686a49e45c11cbf0730e605ba9e7 WatchSource:0}: Error finding container 308356763d43d634bc51b068aec29ba7b814686a49e45c11cbf0730e605ba9e7: Status 404 returned error can't find the container with id 308356763d43d634bc51b068aec29ba7b814686a49e45c11cbf0730e605ba9e7
Sep 16 18:41:51.306313 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.304744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:51.478228 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.477345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:51.479908 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.479883    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:51.492200 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.491956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2b3c86-57c3-4cd2-8142-5be14426c108-metrics-certs\") pod \"network-metrics-daemon-srqqp\" (UID: \"8a2b3c86-57c3-4cd2-8142-5be14426c108\") " pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:51.542656 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.542532    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75ff4865fc-9mq57"]
Sep 16 18:41:51.547779 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:51.547751    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4793c3bc_0e16_4e1b_8ed9_38cc1cd2b154.slice/crio-2ffca0c1deb70e7775487ac231b1882c71762e40d059e1b8eeaf6d6a2510362f WatchSource:0}: Error finding container 2ffca0c1deb70e7775487ac231b1882c71762e40d059e1b8eeaf6d6a2510362f: Status 404 returned error can't find the container with id 2ffca0c1deb70e7775487ac231b1882c71762e40d059e1b8eeaf6d6a2510362f
Sep 16 18:41:51.551351 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.551325    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bf654d699-cbg4l"]
Sep 16 18:41:51.555762 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:51.555730    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7d98d4_c658_4469_8908_5f8610fc8d9f.slice/crio-6e9b3b5e479e19d3f7f4d200bb6530d8d17844a56ffa04e2261cdb17ec98c43d WatchSource:0}: Error finding container 6e9b3b5e479e19d3f7f4d200bb6530d8d17844a56ffa04e2261cdb17ec98c43d: Status 404 returned error can't find the container with id 6e9b3b5e479e19d3f7f4d200bb6530d8d17844a56ffa04e2261cdb17ec98c43d
Sep 16 18:41:51.578599 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.578436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt4p\" (UniqueName: \"kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p\") pod \"network-check-target-7fssr\" (UID: \"664b3dff-845a-40e3-a987-8ff3d897d228\") " pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:51.578599 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.578476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:51.581269 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.581249    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:51.581375 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.581333    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:51.591893 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.591870    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:51.592096 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.592074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b52d6a7-170d-47a1-b771-ff55013146d6-original-pull-secret\") pod \"global-pull-secret-syncer-46l2m\" (UID: \"5b52d6a7-170d-47a1-b771-ff55013146d6\") " pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:51.605822 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.604771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prt4p\" (UniqueName: \"kubernetes.io/projected/664b3dff-845a-40e3-a987-8ff3d897d228-kube-api-access-prt4p\") pod \"network-check-target-7fssr\" (UID: \"664b3dff-845a-40e3-a987-8ff3d897d228\") " pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:51.723847 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.723745    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46l2m"
Sep 16 18:41:51.740905 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.740868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ndh2n\""
Sep 16 18:41:51.748570 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.748543    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srqqp"
Sep 16 18:41:51.748893 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.748877    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5qbcq\""
Sep 16 18:41:51.757371 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:51.757280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:41:52.051057 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.050365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-46l2m"]
Sep 16 18:41:52.053797 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.053721    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:52.076137 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.076098    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:41:52.080200 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.080130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.087740 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.087181    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:41:52.087740 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.087285    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:41:52.087740 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.087529    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:41:52.087740 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.087724    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:41:52.088031 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.087776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:41:52.088031 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.087824    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:41:52.088031 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.087776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-842f39qplcbut\""
Sep 16 18:41:52.088294 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.088084    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:41:52.088350 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.088293    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-srqqp"]
Sep 16 18:41:52.088350 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.088302    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:41:52.088771 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.088449    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:41:52.088771 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.088456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:41:52.088771 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.088551    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:41:52.088771 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.088560    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:41:52.088771 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.088654    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8f6p8\""
Sep 16 18:41:52.092127 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.091295    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:41:52.092127 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.091833    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7fssr"]
Sep 16 18:41:52.095335 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.095315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:41:52.120544 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.117306    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:41:52.132416 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.132346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57" event={"ID":"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154","Type":"ContainerStarted","Data":"2ffca0c1deb70e7775487ac231b1882c71762e40d059e1b8eeaf6d6a2510362f"}
Sep 16 18:41:52.138068 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.138037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5cafeaa-779d-4c84-a719-8af92f10e364","Type":"ContainerStarted","Data":"0ec39b3e069e90affe483238e73fe6430d776529f912bc374cbced406b6c1bdc"}
Sep 16 18:41:52.138244 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.138096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5cafeaa-779d-4c84-a719-8af92f10e364","Type":"ContainerStarted","Data":"3ed695661768e57db08fd441e2d00dcd7433d0d3a1220f4e97d19ec249827b7e"}
Sep 16 18:41:52.138244 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.138123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5cafeaa-779d-4c84-a719-8af92f10e364","Type":"ContainerStarted","Data":"c67d34dc2ac007cd4d9cb24254f327e63db60296dbd54d0a4246f9dfac8fcbd8"}
Sep 16 18:41:52.138244 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.138146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5cafeaa-779d-4c84-a719-8af92f10e364","Type":"ContainerStarted","Data":"45d7c8b3fa2ad94bcb714917285eb0ed0eb30900acb8dec67532befa3c348fc5"}
Sep 16 18:41:52.138244 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.138198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5cafeaa-779d-4c84-a719-8af92f10e364","Type":"ContainerStarted","Data":"0fae290a4dca9289f6e777fb66cd0cf34f100b4a64523bfba921cb2579df56d3"}
Sep 16 18:41:52.144875 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.144840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h" event={"ID":"04aee5ca-0cf7-489a-b9e8-09592e9c12d7","Type":"ContainerStarted","Data":"81a8929f2b77a105833161de30c2c7a024147f4c0636e0fd62184f2602daa5d2"}
Sep 16 18:41:52.144994 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.144925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h" event={"ID":"04aee5ca-0cf7-489a-b9e8-09592e9c12d7","Type":"ContainerStarted","Data":"c575cd274d7227b71077400d5014a2b409196f722afcec680d7b0b6359e34f42"}
Sep 16 18:41:52.144994 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.144968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h" event={"ID":"04aee5ca-0cf7-489a-b9e8-09592e9c12d7","Type":"ContainerStarted","Data":"d20b9279667f058d0000743f921be83b6928fd252fb19f71c1c24df7a6e50fc5"}
Sep 16 18:41:52.147662 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.147635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf654d699-cbg4l" event={"ID":"bc7d98d4-c658-4469-8908-5f8610fc8d9f","Type":"ContainerStarted","Data":"9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561"}
Sep 16 18:41:52.147760 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.147672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf654d699-cbg4l" event={"ID":"bc7d98d4-c658-4469-8908-5f8610fc8d9f","Type":"ContainerStarted","Data":"6e9b3b5e479e19d3f7f4d200bb6530d8d17844a56ffa04e2261cdb17ec98c43d"}
Sep 16 18:41:52.149346 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.149317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64644594b-5c8cb" event={"ID":"b651e8e0-92d6-40b2-8064-7523c2ca41a8","Type":"ContainerStarted","Data":"308356763d43d634bc51b068aec29ba7b814686a49e45c11cbf0730e605ba9e7"}
Sep 16 18:41:52.187751 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.187687    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bf654d699-cbg4l" podStartSLOduration=2.187673167 podStartE2EDuration="2.187673167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:52.187307443 +0000 UTC m=+66.852579894" watchObservedRunningTime="2026-09-16 18:41:52.187673167 +0000 UTC m=+66.852945594"
Sep 16 18:41:52.191645 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.191618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd3074b6-6505-4bd3-8738-213aa47b12c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.191775 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.191698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.191775 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.191745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-config\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.191775 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.191769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.191916 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.191793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.191916 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.191820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r758\" (UniqueName: \"kubernetes.io/projected/dd3074b6-6505-4bd3-8738-213aa47b12c1-kube-api-access-2r758\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.191916 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.191846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.191916 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.191902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192056 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.191928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192056 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.191981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192056 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.192026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192190 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.192090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd3074b6-6505-4bd3-8738-213aa47b12c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192190 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.192116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192190 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.192143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192190 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.192172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192306 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.192212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192306 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.192240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192306 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.192268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192306 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.192290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.192509 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.192316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293044 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293217 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293217 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293217 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293217 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd3074b6-6505-4bd3-8738-213aa47b12c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293217 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293217 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-config\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r758\" (UniqueName: \"kubernetes.io/projected/dd3074b6-6505-4bd3-8738-213aa47b12c1-kube-api-access-2r758\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd3074b6-6505-4bd3-8738-213aa47b12c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293492 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293901 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293901 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.293901 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.293888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.297929 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.295587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.297929 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.296524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.297929 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.296997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.297929 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.297686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd3074b6-6505-4bd3-8738-213aa47b12c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.298238 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.298073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd3074b6-6505-4bd3-8738-213aa47b12c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.298594 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.298569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.299913 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.299664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.302609 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.301339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.302609 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.301954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.302609 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.302530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3074b6-6505-4bd3-8738-213aa47b12c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.302845 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.302691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.304403 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.304345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.304403 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.304382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.304541 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.304507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.304814 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.304789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.306112 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.306067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.306437 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.306416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-config\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.306995 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.306951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd3074b6-6505-4bd3-8738-213aa47b12c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.318875 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.318827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r758\" (UniqueName: \"kubernetes.io/projected/dd3074b6-6505-4bd3-8738-213aa47b12c1-kube-api-access-2r758\") pod \"prometheus-k8s-0\" (UID: \"dd3074b6-6505-4bd3-8738-213aa47b12c1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.386615 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.386577    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:41:52.400230 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:52.400004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:52.596920 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:52.596880    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664b3dff_845a_40e3_a987_8ff3d897d228.slice/crio-d71d09c3dd1f5d54a7c4320616e0bc637530f69a1ada98ff30c6511d58dc692f WatchSource:0}: Error finding container d71d09c3dd1f5d54a7c4320616e0bc637530f69a1ada98ff30c6511d58dc692f: Status 404 returned error can't find the container with id d71d09c3dd1f5d54a7c4320616e0bc637530f69a1ada98ff30c6511d58dc692f
Sep 16 18:41:52.597243 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:52.597212    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2b3c86_57c3_4cd2_8142_5be14426c108.slice/crio-a58899a0ad80133c08918a833e0d3785832d2fba498c67dde66103d774ab2f94 WatchSource:0}: Error finding container a58899a0ad80133c08918a833e0d3785832d2fba498c67dde66103d774ab2f94: Status 404 returned error can't find the container with id a58899a0ad80133c08918a833e0d3785832d2fba498c67dde66103d774ab2f94
Sep 16 18:41:52.598391 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:52.598363    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b52d6a7_170d_47a1_b771_ff55013146d6.slice/crio-db8b473cad31eca91f6d272c830de4964dc415b9762d25188c4dffa4b53c5040 WatchSource:0}: Error finding container db8b473cad31eca91f6d272c830de4964dc415b9762d25188c4dffa4b53c5040: Status 404 returned error can't find the container with id db8b473cad31eca91f6d272c830de4964dc415b9762d25188c4dffa4b53c5040
Sep 16 18:41:53.181231 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:53.173557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-srqqp" event={"ID":"8a2b3c86-57c3-4cd2-8142-5be14426c108","Type":"ContainerStarted","Data":"a58899a0ad80133c08918a833e0d3785832d2fba498c67dde66103d774ab2f94"}
Sep 16 18:41:53.181231 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:53.177391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7fssr" event={"ID":"664b3dff-845a-40e3-a987-8ff3d897d228","Type":"ContainerStarted","Data":"d71d09c3dd1f5d54a7c4320616e0bc637530f69a1ada98ff30c6511d58dc692f"}
Sep 16 18:41:53.182120 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:53.182011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-46l2m" event={"ID":"5b52d6a7-170d-47a1-b771-ff55013146d6","Type":"ContainerStarted","Data":"db8b473cad31eca91f6d272c830de4964dc415b9762d25188c4dffa4b53c5040"}
Sep 16 18:41:53.527135 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:53.527097    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:41:54.191861 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.191821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64644594b-5c8cb" event={"ID":"b651e8e0-92d6-40b2-8064-7523c2ca41a8","Type":"ContainerStarted","Data":"6917f3d1814d09670f58c3cfc1fdbb9a4e9d8c3d296abbd6e7167816aaed1259"}
Sep 16 18:41:54.194303 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.194237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5" event={"ID":"8989cb34-2530-415a-bb8a-1a3542e6cfff","Type":"ContainerStarted","Data":"ec68e6749509f572866bf5cc10cbe38a47881eb07ce072b85e8f46ff4b9588a2"}
Sep 16 18:41:54.195071 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.195048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"
Sep 16 18:41:54.201790 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.201767    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5"
Sep 16 18:41:54.203112 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.202196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5cafeaa-779d-4c84-a719-8af92f10e364","Type":"ContainerStarted","Data":"eb24d1c44bdbaefff0c7455520662a54ba1cfb89a32821fa38f3adfd467853f3"}
Sep 16 18:41:54.208286 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.207484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h" event={"ID":"04aee5ca-0cf7-489a-b9e8-09592e9c12d7","Type":"ContainerStarted","Data":"b10faa9fa55b230e4ca27028b3d52e30f7070fef39f998a26d44e0804fb5c608"}
Sep 16 18:41:54.208286 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.207557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h" event={"ID":"04aee5ca-0cf7-489a-b9e8-09592e9c12d7","Type":"ContainerStarted","Data":"4047740ff7415e20c75cb0d93ff5de80d67b18939ffd3b85c3b4c3037fafad6f"}
Sep 16 18:41:54.208286 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.207600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h" event={"ID":"04aee5ca-0cf7-489a-b9e8-09592e9c12d7","Type":"ContainerStarted","Data":"1e01517c2d787a621014888e93922374b0ee7343b9f4e87c6940dba87089edd4"}
Sep 16 18:41:54.208286 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.207666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:41:54.211928 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.210847    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd3074b6-6505-4bd3-8738-213aa47b12c1" containerID="1694389ed57d94031450c09b4e8067c6130743b935eebd566e049782246824d2" exitCode=0
Sep 16 18:41:54.211928 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.210923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3074b6-6505-4bd3-8738-213aa47b12c1","Type":"ContainerDied","Data":"1694389ed57d94031450c09b4e8067c6130743b935eebd566e049782246824d2"}
Sep 16 18:41:54.211928 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.210984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3074b6-6505-4bd3-8738-213aa47b12c1","Type":"ContainerStarted","Data":"830c10b333f5cc398ec49e6d198724b5e48d159da5e172677e099111c6791a52"}
Sep 16 18:41:54.222996 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.221641    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64644594b-5c8cb" podStartSLOduration=3.33696006 podStartE2EDuration="5.221621084s" totalImagesPullingTime="1.884661024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:54.220754149 +0000 UTC m=+68.886026601" watchObservedRunningTime="2026-09-16 18:41:54.221621084 +0000 UTC m=+68.886893518"
Sep 16 18:41:54.289295 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.289228    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-59pp5" podStartSLOduration=3.025893801 podStartE2EDuration="5.289210438s" totalImagesPullingTime="2.263316637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:54.243497317 +0000 UTC m=+68.908769767" watchObservedRunningTime="2026-09-16 18:41:54.289210438 +0000 UTC m=+68.954482877"
Sep 16 18:41:54.326503 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.325580    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.60233358 podStartE2EDuration="8.325564053s" totalImagesPullingTime="4.704531328s" totalInitContainerRuntime="18.699145ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:54.32379282 +0000 UTC m=+68.989065251" watchObservedRunningTime="2026-09-16 18:41:54.325564053 +0000 UTC m=+68.990836482"
Sep 16 18:41:54.358180 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.358100    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h" podStartSLOduration=3.126164113 podStartE2EDuration="7.358073936s" totalImagesPullingTime="4.231909823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:54.356961281 +0000 UTC m=+69.022233740" watchObservedRunningTime="2026-09-16 18:41:54.358073936 +0000 UTC m=+69.023346366"
Sep 16 18:41:54.740139 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.740099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:54.885032 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:54.884947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:55.239272 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.239216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-srqqp" event={"ID":"8a2b3c86-57c3-4cd2-8142-5be14426c108","Type":"ContainerStarted","Data":"df8a0b2fec4d214101ca7729ef6b234046558afe9cd60e2c560f5948047d32cd"}
Sep 16 18:41:55.239826 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.239301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-srqqp" event={"ID":"8a2b3c86-57c3-4cd2-8142-5be14426c108","Type":"ContainerStarted","Data":"e0a878ab4aa2d809f4918e76c7d795b19a1e0d1c3927886f8f5001b57d61e64a"}
Sep 16 18:41:55.244586 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.244556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57" event={"ID":"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154","Type":"ContainerStarted","Data":"0505b0b5348f94ed086db2c5f8950453038944a05f52272f85c201c3e83f3972"}
Sep 16 18:41:55.244717 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.244619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57" event={"ID":"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154","Type":"ContainerStarted","Data":"0736368a0cb95477e31debe3cb2625d8328a3f1fa141c53868a8580fc5e30249"}
Sep 16 18:41:55.244717 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.244649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57" event={"ID":"4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154","Type":"ContainerStarted","Data":"96f25ab39c7a5c511be4de422e431159efce69e9f7173658408f6bd61c2e0a02"}
Sep 16 18:41:55.304432 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.304304    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-srqqp" podStartSLOduration=67.097231893 podStartE2EDuration="1m9.304282165s" totalImagesPullingTime="2.207050272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:55.270387039 +0000 UTC m=+69.935659486" watchObservedRunningTime="2026-09-16 18:41:55.304282165 +0000 UTC m=+69.969554596"
Sep 16 18:41:55.305305 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.305254    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75ff4865fc-9mq57" podStartSLOduration=2.0260389070000002 podStartE2EDuration="5.305238637s" totalImagesPullingTime="3.27919973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:55.303231819 +0000 UTC m=+69.968504275" watchObservedRunningTime="2026-09-16 18:41:55.305238637 +0000 UTC m=+69.970511065"
Sep 16 18:41:55.447892 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.447851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:55.518047 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.517260    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bf654d699-cbg4l"]
Sep 16 18:41:55.579080 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.579042    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c668cd5bf-2bfz6"]
Sep 16 18:41:55.584218 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.584191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.629734 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.629680    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c668cd5bf-2bfz6"]
Sep 16 18:41:55.629892 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.629867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:55.632882 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.630142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-trusted-ca-bundle\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.632882 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.630241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-oauth-serving-cert\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.632882 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.630280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-oauth-config\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.632882 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.630365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-serving-cert\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.632882 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.630411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbz5l\" (UniqueName: \"kubernetes.io/projected/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-kube-api-access-tbz5l\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.632882 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.630449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-service-ca\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.632882 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.630489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-config\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.731670 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.731623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-trusted-ca-bundle\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.731854 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.731681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-oauth-serving-cert\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.731854 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.731708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-oauth-config\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.731854 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.731759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-serving-cert\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.731854 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.731804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbz5l\" (UniqueName: \"kubernetes.io/projected/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-kube-api-access-tbz5l\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.731854 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.731839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-service-ca\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.732054 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.731864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-config\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.732601 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.732576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-config\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.732729 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.732667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-trusted-ca-bundle\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.733045 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.733022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-oauth-serving-cert\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.733610 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.733586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-service-ca\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.735805 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.735780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-serving-cert\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.736237 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.736215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-oauth-config\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.745331 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.745297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbz5l\" (UniqueName: \"kubernetes.io/projected/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-kube-api-access-tbz5l\") pod \"console-6c668cd5bf-2bfz6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") " pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.790501 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.790459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:55.899525 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.899441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:41:55.905120 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:55.905094    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:56.001799 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:56.001762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:59.719183 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:41:59.719131    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c668cd5bf-2bfz6"]
Sep 16 18:41:59.723852 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:41:59.723818    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e85b4fb_b7d5_4a5c_9088_ae6ab44ba7c6.slice/crio-51a5f509485ab4bcd87aa3359e1d78939174409faa668ba5d6acc9005359cb7d WatchSource:0}: Error finding container 51a5f509485ab4bcd87aa3359e1d78939174409faa668ba5d6acc9005359cb7d: Status 404 returned error can't find the container with id 51a5f509485ab4bcd87aa3359e1d78939174409faa668ba5d6acc9005359cb7d
Sep 16 18:42:00.250348 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.250317    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9b49dff5d-g777h"
Sep 16 18:42:00.269034 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.268999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7fssr" event={"ID":"664b3dff-845a-40e3-a987-8ff3d897d228","Type":"ContainerStarted","Data":"4fae5ad1971581d84f28f135ff5185ea679efadc3c51a0b908e92712c9aecc03"}
Sep 16 18:42:00.269230 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.269096    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:42:00.270491 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.270468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-46l2m" event={"ID":"5b52d6a7-170d-47a1-b771-ff55013146d6","Type":"ContainerStarted","Data":"937a1d612d11078a0836b6c3378472f46d31cdbbb0410f45846c6d5d0ca1ae23"}
Sep 16 18:42:00.273843 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.273824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3074b6-6505-4bd3-8738-213aa47b12c1","Type":"ContainerStarted","Data":"767bf412edce6bb02f143957a2bb8a8e0211f3fbfc58a0973e04c3a2e996dfde"}
Sep 16 18:42:00.273929 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.273868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3074b6-6505-4bd3-8738-213aa47b12c1","Type":"ContainerStarted","Data":"768858771ad0f0ca554d982fcd499fa404ede36bd72719532e0ffe17a1b749aa"}
Sep 16 18:42:00.273929 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.273911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3074b6-6505-4bd3-8738-213aa47b12c1","Type":"ContainerStarted","Data":"1b8be9d941d482afce61c0d0583cd376e78b6abd30b8e83c0210aa3c0709d182"}
Sep 16 18:42:00.274011 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.273942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3074b6-6505-4bd3-8738-213aa47b12c1","Type":"ContainerStarted","Data":"7456afa6294ff3811660f263d0221ab23e201e4b010a02ef897a97c1d2acc500"}
Sep 16 18:42:00.274011 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.273966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3074b6-6505-4bd3-8738-213aa47b12c1","Type":"ContainerStarted","Data":"8a4ce00484ab69a8fbfa08bfc75bfc5c78e6e5b8acc8b81e180707ab59c7e5d8"}
Sep 16 18:42:00.274011 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.273990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3074b6-6505-4bd3-8738-213aa47b12c1","Type":"ContainerStarted","Data":"3e54b2347b12b90fd5d66eceb8ffb170da620fa06e0363f0dd3f4ef0a77c700e"}
Sep 16 18:42:00.275581 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.275564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c668cd5bf-2bfz6" event={"ID":"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6","Type":"ContainerStarted","Data":"ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e"}
Sep 16 18:42:00.275655 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.275591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c668cd5bf-2bfz6" event={"ID":"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6","Type":"ContainerStarted","Data":"51a5f509485ab4bcd87aa3359e1d78939174409faa668ba5d6acc9005359cb7d"}
Sep 16 18:42:00.311406 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.311341    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c668cd5bf-2bfz6" podStartSLOduration=5.311320547 podStartE2EDuration="5.311320547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:00.306489824 +0000 UTC m=+74.971762280" watchObservedRunningTime="2026-09-16 18:42:00.311320547 +0000 UTC m=+74.976592978"
Sep 16 18:42:00.352726 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.352663    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.105009435 podStartE2EDuration="8.352646507s" totalImagesPullingTime="5.220717935s" totalInitContainerRuntime="26.919137ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:00.351405116 +0000 UTC m=+75.016677545" watchObservedRunningTime="2026-09-16 18:42:00.352646507 +0000 UTC m=+75.017918936"
Sep 16 18:42:00.380991 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.380936    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7fssr" podStartSLOduration=67.569966621 podStartE2EDuration="1m14.380921932s" totalImagesPullingTime="6.810955311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:00.379401331 +0000 UTC m=+75.044673781" watchObservedRunningTime="2026-09-16 18:42:00.380921932 +0000 UTC m=+75.046194361"
Sep 16 18:42:00.403388 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.403343    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-46l2m" podStartSLOduration=67.579378801 podStartE2EDuration="1m14.40332639s" totalImagesPullingTime="6.823947589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:00.401715001 +0000 UTC m=+75.066987463" watchObservedRunningTime="2026-09-16 18:42:00.40332639 +0000 UTC m=+75.068598821"
Sep 16 18:42:00.524612 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.524532    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:00.611472 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.611442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:00.881179 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:00.881137    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:42:02.400406 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:02.400366    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:03.223328 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:03.223293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:03.334271 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:03.334239    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:03.767593 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:03.767562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:03.852321 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:03.852291    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:05.899947 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:05.899918    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:42:05.900330 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:05.899967    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:42:05.904653 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:05.904634    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:42:06.309540 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:06.309467    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:42:06.527330 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:06.527291    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:06.797092 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:06.797059    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:09.382100 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:09.382067    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:09.578709 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:09.578677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:09.670898 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:09.670826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:10.108411 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:10.108375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:42:10.108574 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:10.108433    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:42:15.527630 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.527566    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f7f88989c-9rr5p" podUID="a6f88f9d-22aa-4985-b319-bfdef20ed5de" containerName="console" containerID="cri-o://348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2" gracePeriod=15
Sep 16 18:42:15.777067 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.777040    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:42:15.930296 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930255    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-config\") pod \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") "
Sep 16 18:42:15.930455 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cc6v\" (UniqueName: \"kubernetes.io/projected/a6f88f9d-22aa-4985-b319-bfdef20ed5de-kube-api-access-8cc6v\") pod \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") "
Sep 16 18:42:15.930455 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-service-ca\") pod \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") "
Sep 16 18:42:15.930455 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-trusted-ca-bundle\") pod \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") "
Sep 16 18:42:15.930455 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930384    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-serving-cert\") pod \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") "
Sep 16 18:42:15.930455 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930444    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-oauth-config\") pod \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") "
Sep 16 18:42:15.930729 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930496    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-oauth-serving-cert\") pod \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\" (UID: \"a6f88f9d-22aa-4985-b319-bfdef20ed5de\") "
Sep 16 18:42:15.930729 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-config" (OuterVolumeSpecName: "console-config") pod "a6f88f9d-22aa-4985-b319-bfdef20ed5de" (UID: "a6f88f9d-22aa-4985-b319-bfdef20ed5de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:15.930829 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930805    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a6f88f9d-22aa-4985-b319-bfdef20ed5de" (UID: "a6f88f9d-22aa-4985-b319-bfdef20ed5de"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:15.930872 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930862    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:15.931015 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.930912    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a6f88f9d-22aa-4985-b319-bfdef20ed5de" (UID: "a6f88f9d-22aa-4985-b319-bfdef20ed5de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:15.931092 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.931071    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-service-ca" (OuterVolumeSpecName: "service-ca") pod "a6f88f9d-22aa-4985-b319-bfdef20ed5de" (UID: "a6f88f9d-22aa-4985-b319-bfdef20ed5de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:15.932979 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.932947    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f88f9d-22aa-4985-b319-bfdef20ed5de-kube-api-access-8cc6v" (OuterVolumeSpecName: "kube-api-access-8cc6v") pod "a6f88f9d-22aa-4985-b319-bfdef20ed5de" (UID: "a6f88f9d-22aa-4985-b319-bfdef20ed5de"). InnerVolumeSpecName "kube-api-access-8cc6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:15.932979 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.932951    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a6f88f9d-22aa-4985-b319-bfdef20ed5de" (UID: "a6f88f9d-22aa-4985-b319-bfdef20ed5de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:15.933109 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:15.933010    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a6f88f9d-22aa-4985-b319-bfdef20ed5de" (UID: "a6f88f9d-22aa-4985-b319-bfdef20ed5de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:16.032196 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.032144    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cc6v\" (UniqueName: \"kubernetes.io/projected/a6f88f9d-22aa-4985-b319-bfdef20ed5de-kube-api-access-8cc6v\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:16.032196 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.032192    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-service-ca\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:16.032196 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.032201    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-trusted-ca-bundle\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:16.032196 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.032209    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-serving-cert\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:16.032484 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.032218    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6f88f9d-22aa-4985-b319-bfdef20ed5de-console-oauth-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:16.032484 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.032228    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6f88f9d-22aa-4985-b319-bfdef20ed5de-oauth-serving-cert\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:16.349789 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.349753    2579 generic.go:356] "Generic (PLEG): container finished" podID="a6f88f9d-22aa-4985-b319-bfdef20ed5de" containerID="348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2" exitCode=0
Sep 16 18:42:16.349935 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.349797    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f88989c-9rr5p"
Sep 16 18:42:16.349935 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.349832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f88989c-9rr5p" event={"ID":"a6f88f9d-22aa-4985-b319-bfdef20ed5de","Type":"ContainerDied","Data":"348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2"}
Sep 16 18:42:16.349935 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.349881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f88989c-9rr5p" event={"ID":"a6f88f9d-22aa-4985-b319-bfdef20ed5de","Type":"ContainerDied","Data":"f1bde6bbd66f5a28fbc9301d3efd45c0a57f810e68de7938480ef691a3e9f540"}
Sep 16 18:42:16.349935 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.349899    2579 scope.go:117] "RemoveContainer" containerID="348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2"
Sep 16 18:42:16.361545 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.361379    2579 scope.go:117] "RemoveContainer" containerID="348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2"
Sep 16 18:42:16.361812 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:42:16.361769    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2\": container with ID starting with 348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2 not found: ID does not exist" containerID="348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2"
Sep 16 18:42:16.361864 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.361826    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2"} err="failed to get container status \"348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2\": rpc error: code = NotFound desc = could not find container \"348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2\": container with ID starting with 348be8a285fa4121021eb5c9c3631086a4593a4493f496b0fefe58e8e4a460b2 not found: ID does not exist"
Sep 16 18:42:16.373531 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.373503    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7f88989c-9rr5p"]
Sep 16 18:42:16.379499 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:16.379477    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f7f88989c-9rr5p"]
Sep 16 18:42:17.811420 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:17.811381    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f88f9d-22aa-4985-b319-bfdef20ed5de" path="/var/lib/kubelet/pods/a6f88f9d-22aa-4985-b319-bfdef20ed5de/volumes"
Sep 16 18:42:18.791367 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:18.791328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:18.874110 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:18.874074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:20.548125 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.548075    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bf654d699-cbg4l" podUID="bc7d98d4-c658-4469-8908-5f8610fc8d9f" containerName="console" containerID="cri-o://9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561" gracePeriod=15
Sep 16 18:42:20.809868 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.809800    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:42:20.977309 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977274    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-oauth-config\") pod \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") "
Sep 16 18:42:20.977309 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcrj4\" (UniqueName: \"kubernetes.io/projected/bc7d98d4-c658-4469-8908-5f8610fc8d9f-kube-api-access-tcrj4\") pod \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") "
Sep 16 18:42:20.977556 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977333    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-service-ca\") pod \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") "
Sep 16 18:42:20.977556 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-config\") pod \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") "
Sep 16 18:42:20.977556 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-trusted-ca-bundle\") pod \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") "
Sep 16 18:42:20.977712 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977616    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-oauth-serving-cert\") pod \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") "
Sep 16 18:42:20.977712 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977657    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-serving-cert\") pod \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\" (UID: \"bc7d98d4-c658-4469-8908-5f8610fc8d9f\") "
Sep 16 18:42:20.977872 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-service-ca" (OuterVolumeSpecName: "service-ca") pod "bc7d98d4-c658-4469-8908-5f8610fc8d9f" (UID: "bc7d98d4-c658-4469-8908-5f8610fc8d9f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:20.977987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977962    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bc7d98d4-c658-4469-8908-5f8610fc8d9f" (UID: "bc7d98d4-c658-4469-8908-5f8610fc8d9f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:20.978050 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977980    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-service-ca\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:20.978050 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.977992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bc7d98d4-c658-4469-8908-5f8610fc8d9f" (UID: "bc7d98d4-c658-4469-8908-5f8610fc8d9f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:20.978344 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.978055    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-config" (OuterVolumeSpecName: "console-config") pod "bc7d98d4-c658-4469-8908-5f8610fc8d9f" (UID: "bc7d98d4-c658-4469-8908-5f8610fc8d9f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:20.979850 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.979821    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bc7d98d4-c658-4469-8908-5f8610fc8d9f" (UID: "bc7d98d4-c658-4469-8908-5f8610fc8d9f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:20.979951 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.979852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7d98d4-c658-4469-8908-5f8610fc8d9f-kube-api-access-tcrj4" (OuterVolumeSpecName: "kube-api-access-tcrj4") pod "bc7d98d4-c658-4469-8908-5f8610fc8d9f" (UID: "bc7d98d4-c658-4469-8908-5f8610fc8d9f"). InnerVolumeSpecName "kube-api-access-tcrj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:20.979951 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:20.979862    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bc7d98d4-c658-4469-8908-5f8610fc8d9f" (UID: "bc7d98d4-c658-4469-8908-5f8610fc8d9f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:21.078630 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.078604    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-trusted-ca-bundle\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:21.078630 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.078631    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-oauth-serving-cert\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:21.078804 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.078643    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-serving-cert\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:21.078804 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.078652    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-oauth-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:21.078804 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.078661    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcrj4\" (UniqueName: \"kubernetes.io/projected/bc7d98d4-c658-4469-8908-5f8610fc8d9f-kube-api-access-tcrj4\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:21.078804 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.078669    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc7d98d4-c658-4469-8908-5f8610fc8d9f-console-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:42:21.374519 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.374429    2579 generic.go:356] "Generic (PLEG): container finished" podID="bc7d98d4-c658-4469-8908-5f8610fc8d9f" containerID="9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561" exitCode=0
Sep 16 18:42:21.374519 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.374479    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf654d699-cbg4l"
Sep 16 18:42:21.374735 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.374518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf654d699-cbg4l" event={"ID":"bc7d98d4-c658-4469-8908-5f8610fc8d9f","Type":"ContainerDied","Data":"9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561"}
Sep 16 18:42:21.374735 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.374566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf654d699-cbg4l" event={"ID":"bc7d98d4-c658-4469-8908-5f8610fc8d9f","Type":"ContainerDied","Data":"6e9b3b5e479e19d3f7f4d200bb6530d8d17844a56ffa04e2261cdb17ec98c43d"}
Sep 16 18:42:21.374735 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.374587    2579 scope.go:117] "RemoveContainer" containerID="9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561"
Sep 16 18:42:21.383713 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.383695    2579 scope.go:117] "RemoveContainer" containerID="9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561"
Sep 16 18:42:21.383995 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:42:21.383975    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561\": container with ID starting with 9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561 not found: ID does not exist" containerID="9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561"
Sep 16 18:42:21.384045 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.384004    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561"} err="failed to get container status \"9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561\": rpc error: code = NotFound desc = could not find container \"9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561\": container with ID starting with 9fd44412449c0f20e84daf0bf1a4425bb3dc7978cc3ad902d6e86a05908a8561 not found: ID does not exist"
Sep 16 18:42:21.401914 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.401889    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bf654d699-cbg4l"]
Sep 16 18:42:21.411221 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.411199    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bf654d699-cbg4l"]
Sep 16 18:42:21.811411 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:21.811327    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7d98d4-c658-4469-8908-5f8610fc8d9f" path="/var/lib/kubelet/pods/bc7d98d4-c658-4469-8908-5f8610fc8d9f/volumes"
Sep 16 18:42:27.872546 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:27.872505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:27.972450 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:27.972419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:30.113552 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:30.113522    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:42:30.117222 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:30.117200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64644594b-5c8cb"
Sep 16 18:42:31.282952 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:31.282919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7fssr"
Sep 16 18:42:52.400533 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:52.400500    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:52.420599 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:52.420570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:52.525975 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:52.525948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:58.115653 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:42:58.115618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:01.768907 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:01.768872    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:01.835891 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:01.835864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:02.512764 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:02.512728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:02.584322 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:02.584290    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:12.868265 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:12.868228    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c668cd5bf-2bfz6"]
Sep 16 18:43:12.990002 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:12.989974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:13.090593 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:13.090561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:13.157066 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:13.157004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:13.743777 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:13.743744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:13.813326 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:13.813300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:18.765389 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:18.765355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:18.864254 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:18.864221    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:18.964815 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:18.964782    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:28.987557 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:28.987520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:37.887739 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:37.887654    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c668cd5bf-2bfz6" podUID="8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" containerName="console" containerID="cri-o://ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e" gracePeriod=15
Sep 16 18:43:38.136617 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.136594    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:43:38.208993 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.208910    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-oauth-config\") pod \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") "
Sep 16 18:43:38.208993 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.208951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-config\") pod \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") "
Sep 16 18:43:38.208993 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.208981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-service-ca\") pod \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") "
Sep 16 18:43:38.209287 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.209065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-oauth-serving-cert\") pod \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") "
Sep 16 18:43:38.209287 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.209094    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-trusted-ca-bundle\") pod \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") "
Sep 16 18:43:38.209287 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.209140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-serving-cert\") pod \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") "
Sep 16 18:43:38.209287 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.209220    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbz5l\" (UniqueName: \"kubernetes.io/projected/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-kube-api-access-tbz5l\") pod \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\" (UID: \"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6\") "
Sep 16 18:43:38.209549 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.209395    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-config" (OuterVolumeSpecName: "console-config") pod "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" (UID: "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:38.209549 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.209415    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-service-ca" (OuterVolumeSpecName: "service-ca") pod "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" (UID: "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:38.209935 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.209762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" (UID: "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:38.209935 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.209828    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" (UID: "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:38.211570 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.211537    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" (UID: "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:38.211660 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.211575    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-kube-api-access-tbz5l" (OuterVolumeSpecName: "kube-api-access-tbz5l") pod "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" (UID: "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6"). InnerVolumeSpecName "kube-api-access-tbz5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:38.211660 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.211584    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" (UID: "8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:38.310427 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.310390    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-oauth-serving-cert\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:43:38.310427 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.310420    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-trusted-ca-bundle\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:43:38.310427 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.310430    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-serving-cert\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:43:38.310661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.310447    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbz5l\" (UniqueName: \"kubernetes.io/projected/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-kube-api-access-tbz5l\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:43:38.310661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.310456    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-oauth-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:43:38.310661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.310465    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-console-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:43:38.310661 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.310475    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6-service-ca\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:43:38.691217 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.691181    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" containerID="ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e" exitCode=0
Sep 16 18:43:38.691413 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.691255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c668cd5bf-2bfz6" event={"ID":"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6","Type":"ContainerDied","Data":"ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e"}
Sep 16 18:43:38.691413 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.691280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c668cd5bf-2bfz6"
Sep 16 18:43:38.691413 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.691290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c668cd5bf-2bfz6" event={"ID":"8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6","Type":"ContainerDied","Data":"51a5f509485ab4bcd87aa3359e1d78939174409faa668ba5d6acc9005359cb7d"}
Sep 16 18:43:38.691413 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.691309    2579 scope.go:117] "RemoveContainer" containerID="ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e"
Sep 16 18:43:38.700648 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.700631    2579 scope.go:117] "RemoveContainer" containerID="ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e"
Sep 16 18:43:38.700934 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:43:38.700914    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e\": container with ID starting with ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e not found: ID does not exist" containerID="ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e"
Sep 16 18:43:38.700983 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.700945    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e"} err="failed to get container status \"ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e\": rpc error: code = NotFound desc = could not find container \"ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e\": container with ID starting with ce9bc15baf201e9b2579d9f65c4bd9bd2961e55b8550a4fd667eed5184d5951e not found: ID does not exist"
Sep 16 18:43:38.718715 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.718679    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c668cd5bf-2bfz6"]
Sep 16 18:43:38.728188 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:38.728145    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c668cd5bf-2bfz6"]
Sep 16 18:43:39.811270 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:39.811234    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" path="/var/lib/kubelet/pods/8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6/volumes"
Sep 16 18:43:58.592808 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:43:58.592772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:21.567144 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:44:21.567070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.599120 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:44:41.599071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.701024 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:44:41.700945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.799980 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:44:41.799949    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:07.125324 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:07.125245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:07.214729 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:07.214698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.217016 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:14.216985    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.532757 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:14.532672    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.870689 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:14.870657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:15.492443 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:15.492409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:15.685392 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:15.685357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:15.784923 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:15.784846    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:15.892126 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:15.892098    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:16.288783 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:16.288694    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:16.474455 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:16.474419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:16.574465 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:16.574431    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:16.681646 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:16.681614    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.694900 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:45:45.694870    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:46:08.511574 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:08.511541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.424744 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:10.424705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.631468 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:40.631382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.717840 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:40.717812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.841959 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:40.841920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.968761 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:40.968669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:41.072312 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:41.072281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:41.381404 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:41.381370    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:41.482629 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:41.482596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:41.587780 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:41.587746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.595461 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:44.595422    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.658894 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:44.658865    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.736365 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:44.736332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:45.320920 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:45.320883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:45.393429 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:45.393392    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:47.007686 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:47.007649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:47.116759 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:47.116724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:56.002427 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:56.002390    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:56.101964 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:56.101933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:56.169369 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:46:56.169336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:01.202956 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:47:01.202924    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:02.340135 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.340095    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:02.386555 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.386520    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-597x6"]
Sep 16 18:48:02.386873 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.386860    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f88f9d-22aa-4985-b319-bfdef20ed5de" containerName="console"
Sep 16 18:48:02.386933 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.386875    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f88f9d-22aa-4985-b319-bfdef20ed5de" containerName="console"
Sep 16 18:48:02.386933 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.386884    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7d98d4-c658-4469-8908-5f8610fc8d9f" containerName="console"
Sep 16 18:48:02.386933 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.386889    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7d98d4-c658-4469-8908-5f8610fc8d9f" containerName="console"
Sep 16 18:48:02.386933 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.386902    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" containerName="console"
Sep 16 18:48:02.386933 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.386909    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" containerName="console"
Sep 16 18:48:02.387100 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.386964    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e85b4fb-b7d5-4a5c-9088-ae6ab44ba7c6" containerName="console"
Sep 16 18:48:02.387100 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.386971    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f88f9d-22aa-4985-b319-bfdef20ed5de" containerName="console"
Sep 16 18:48:02.387100 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.386981    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7d98d4-c658-4469-8908-5f8610fc8d9f" containerName="console"
Sep 16 18:48:02.390807 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.390786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:02.394036 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.394017    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-zrc79\""
Sep 16 18:48:02.394726 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.394709    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:48:02.394935 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.394919    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 18:48:02.395017 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.394964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:48:02.403742 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.403718    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-597x6"]
Sep 16 18:48:02.406020 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.405997    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-4lnvt"]
Sep 16 18:48:02.408275 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.408259    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.412573 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.412539    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:48:02.412688 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.412540    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-wjdpj\""
Sep 16 18:48:02.412804 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.412787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:48:02.422805 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.422784    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-4lnvt"]
Sep 16 18:48:02.447920 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.447895    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:02.480564 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.480531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ea1a68-d91a-4c19-8ead-19bbf99edc8b-metrics-certs\") pod \"odh-model-controller-7b966d7984-4lnvt\" (UID: \"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b\") " pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.480772 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.480752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tdzn\" (UniqueName: \"kubernetes.io/projected/63fe53c4-2c29-4698-8cc4-bc7d13210fb1-kube-api-access-6tdzn\") pod \"model-serving-api-7b954ffc95-597x6\" (UID: \"63fe53c4-2c29-4698-8cc4-bc7d13210fb1\") " pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:02.480913 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.480899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63fe53c4-2c29-4698-8cc4-bc7d13210fb1-tls-certs\") pod \"model-serving-api-7b954ffc95-597x6\" (UID: \"63fe53c4-2c29-4698-8cc4-bc7d13210fb1\") " pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:02.481009 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.480997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5ea1a68-d91a-4c19-8ead-19bbf99edc8b-cert\") pod \"odh-model-controller-7b966d7984-4lnvt\" (UID: \"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b\") " pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.481124 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.481112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5n5z\" (UniqueName: \"kubernetes.io/projected/e5ea1a68-d91a-4c19-8ead-19bbf99edc8b-kube-api-access-g5n5z\") pod \"odh-model-controller-7b966d7984-4lnvt\" (UID: \"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b\") " pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.552804 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.552772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:02.582562 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.582538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ea1a68-d91a-4c19-8ead-19bbf99edc8b-metrics-certs\") pod \"odh-model-controller-7b966d7984-4lnvt\" (UID: \"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b\") " pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.582726 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.582568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tdzn\" (UniqueName: \"kubernetes.io/projected/63fe53c4-2c29-4698-8cc4-bc7d13210fb1-kube-api-access-6tdzn\") pod \"model-serving-api-7b954ffc95-597x6\" (UID: \"63fe53c4-2c29-4698-8cc4-bc7d13210fb1\") " pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:02.582726 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.582704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63fe53c4-2c29-4698-8cc4-bc7d13210fb1-tls-certs\") pod \"model-serving-api-7b954ffc95-597x6\" (UID: \"63fe53c4-2c29-4698-8cc4-bc7d13210fb1\") " pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:02.582845 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.582744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5ea1a68-d91a-4c19-8ead-19bbf99edc8b-cert\") pod \"odh-model-controller-7b966d7984-4lnvt\" (UID: \"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b\") " pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.582845 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.582785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5n5z\" (UniqueName: \"kubernetes.io/projected/e5ea1a68-d91a-4c19-8ead-19bbf99edc8b-kube-api-access-g5n5z\") pod \"odh-model-controller-7b966d7984-4lnvt\" (UID: \"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b\") " pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.582845 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:48:02.582788    2579 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 18:48:02.582999 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:48:02.582862    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63fe53c4-2c29-4698-8cc4-bc7d13210fb1-tls-certs podName:63fe53c4-2c29-4698-8cc4-bc7d13210fb1 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:03.082841243 +0000 UTC m=+437.748113667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/63fe53c4-2c29-4698-8cc4-bc7d13210fb1-tls-certs") pod "model-serving-api-7b954ffc95-597x6" (UID: "63fe53c4-2c29-4698-8cc4-bc7d13210fb1") : secret "model-serving-api-tls" not found
Sep 16 18:48:02.585022 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.585001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ea1a68-d91a-4c19-8ead-19bbf99edc8b-metrics-certs\") pod \"odh-model-controller-7b966d7984-4lnvt\" (UID: \"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b\") " pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.585114 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.585045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5ea1a68-d91a-4c19-8ead-19bbf99edc8b-cert\") pod \"odh-model-controller-7b966d7984-4lnvt\" (UID: \"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b\") " pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.592708 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.592658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tdzn\" (UniqueName: \"kubernetes.io/projected/63fe53c4-2c29-4698-8cc4-bc7d13210fb1-kube-api-access-6tdzn\") pod \"model-serving-api-7b954ffc95-597x6\" (UID: \"63fe53c4-2c29-4698-8cc4-bc7d13210fb1\") " pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:02.592829 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.592812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5n5z\" (UniqueName: \"kubernetes.io/projected/e5ea1a68-d91a-4c19-8ead-19bbf99edc8b-kube-api-access-g5n5z\") pod \"odh-model-controller-7b966d7984-4lnvt\" (UID: \"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b\") " pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.718527 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.718464    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:02.874750 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.874676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:02.989933 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.989912    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-4lnvt"]
Sep 16 18:48:02.992538 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:48:02.992515    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ea1a68_d91a_4c19_8ead_19bbf99edc8b.slice/crio-2d87bc6c11151ac8ccc3f95db3bc9c166ef6573908f5d4d7adf7de669628d09e WatchSource:0}: Error finding container 2d87bc6c11151ac8ccc3f95db3bc9c166ef6573908f5d4d7adf7de669628d09e: Status 404 returned error can't find the container with id 2d87bc6c11151ac8ccc3f95db3bc9c166ef6573908f5d4d7adf7de669628d09e
Sep 16 18:48:02.993692 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:02.993675    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:48:03.087794 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:03.087758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63fe53c4-2c29-4698-8cc4-bc7d13210fb1-tls-certs\") pod \"model-serving-api-7b954ffc95-597x6\" (UID: \"63fe53c4-2c29-4698-8cc4-bc7d13210fb1\") " pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:03.090079 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:03.090061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63fe53c4-2c29-4698-8cc4-bc7d13210fb1-tls-certs\") pod \"model-serving-api-7b954ffc95-597x6\" (UID: \"63fe53c4-2c29-4698-8cc4-bc7d13210fb1\") " pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:03.301108 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:03.301032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:03.574879 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:03.574857    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-597x6"]
Sep 16 18:48:03.577099 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:48:03.577071    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fe53c4_2c29_4698_8cc4_bc7d13210fb1.slice/crio-d1ccf178f096d96ed7f9866aca9a720f5afaf37cb1ed6ab0a257fa50fd615d48 WatchSource:0}: Error finding container d1ccf178f096d96ed7f9866aca9a720f5afaf37cb1ed6ab0a257fa50fd615d48: Status 404 returned error can't find the container with id d1ccf178f096d96ed7f9866aca9a720f5afaf37cb1ed6ab0a257fa50fd615d48
Sep 16 18:48:03.695354 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:03.695321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:03.729379 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:03.729329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-4lnvt" event={"ID":"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b","Type":"ContainerStarted","Data":"2d87bc6c11151ac8ccc3f95db3bc9c166ef6573908f5d4d7adf7de669628d09e"}
Sep 16 18:48:03.730873 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:03.730844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-597x6" event={"ID":"63fe53c4-2c29-4698-8cc4-bc7d13210fb1","Type":"ContainerStarted","Data":"d1ccf178f096d96ed7f9866aca9a720f5afaf37cb1ed6ab0a257fa50fd615d48"}
Sep 16 18:48:03.796754 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:03.796713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:06.750700 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:06.750672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-4lnvt" event={"ID":"e5ea1a68-d91a-4c19-8ead-19bbf99edc8b","Type":"ContainerStarted","Data":"6b3aab80dcc3faf4e43bb00af6cac9a3fb647eb15bd7200e0457fb0b64036d02"}
Sep 16 18:48:06.751090 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:06.750805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:06.752463 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:06.752438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-597x6" event={"ID":"63fe53c4-2c29-4698-8cc4-bc7d13210fb1","Type":"ContainerStarted","Data":"7da8abebcfe18115407441d6a929af3818151d5d2069aa7a1d7684f73986261a"}
Sep 16 18:48:06.752594 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:06.752485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:06.776431 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:06.776389    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-4lnvt" podStartSLOduration=1.092550536 podStartE2EDuration="4.776375022s" totalImagesPullingTime="3.683824486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:06.772893823 +0000 UTC m=+441.438166272" watchObservedRunningTime="2026-09-16 18:48:06.776375022 +0000 UTC m=+441.441647466"
Sep 16 18:48:06.796444 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:06.796393    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-597x6" podStartSLOduration=1.689002007 podStartE2EDuration="4.796379288s" totalImagesPullingTime="3.107377281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:06.794560498 +0000 UTC m=+441.459832948" watchObservedRunningTime="2026-09-16 18:48:06.796379288 +0000 UTC m=+441.461651720"
Sep 16 18:48:17.758285 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:17.758254    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-4lnvt"
Sep 16 18:48:17.760420 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:17.760400    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-597x6"
Sep 16 18:48:18.635735 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.635699    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-zp7gn"]
Sep 16 18:48:18.639178 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.639133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zp7gn"
Sep 16 18:48:18.642128 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.642107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:48:18.642306 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.642287    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:48:18.642358 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.642303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-znrv2\""
Sep 16 18:48:18.655169 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.655132    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zp7gn"]
Sep 16 18:48:18.723040 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.723016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec391b92-7e84-445b-9344-ba99b3adc52d-model-cache\") pod \"s3-init-zp7gn\" (UID: \"ec391b92-7e84-445b-9344-ba99b3adc52d\") " pod="kserve/s3-init-zp7gn"
Sep 16 18:48:18.723135 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.723042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdmhz\" (UniqueName: \"kubernetes.io/projected/ec391b92-7e84-445b-9344-ba99b3adc52d-kube-api-access-cdmhz\") pod \"s3-init-zp7gn\" (UID: \"ec391b92-7e84-445b-9344-ba99b3adc52d\") " pod="kserve/s3-init-zp7gn"
Sep 16 18:48:18.824353 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.824327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec391b92-7e84-445b-9344-ba99b3adc52d-model-cache\") pod \"s3-init-zp7gn\" (UID: \"ec391b92-7e84-445b-9344-ba99b3adc52d\") " pod="kserve/s3-init-zp7gn"
Sep 16 18:48:18.824636 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.824354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdmhz\" (UniqueName: \"kubernetes.io/projected/ec391b92-7e84-445b-9344-ba99b3adc52d-kube-api-access-cdmhz\") pod \"s3-init-zp7gn\" (UID: \"ec391b92-7e84-445b-9344-ba99b3adc52d\") " pod="kserve/s3-init-zp7gn"
Sep 16 18:48:18.824746 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.824730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec391b92-7e84-445b-9344-ba99b3adc52d-model-cache\") pod \"s3-init-zp7gn\" (UID: \"ec391b92-7e84-445b-9344-ba99b3adc52d\") " pod="kserve/s3-init-zp7gn"
Sep 16 18:48:18.837862 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.837836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdmhz\" (UniqueName: \"kubernetes.io/projected/ec391b92-7e84-445b-9344-ba99b3adc52d-kube-api-access-cdmhz\") pod \"s3-init-zp7gn\" (UID: \"ec391b92-7e84-445b-9344-ba99b3adc52d\") " pod="kserve/s3-init-zp7gn"
Sep 16 18:48:18.961482 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:18.961430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zp7gn"
Sep 16 18:48:19.239618 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:19.239547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zp7gn"]
Sep 16 18:48:19.243747 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:48:19.243721    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec391b92_7e84_445b_9344_ba99b3adc52d.slice/crio-ddbcbf1e5800200a9a59353d282e7f1eb762961b802810b437bd96fcd5500955 WatchSource:0}: Error finding container ddbcbf1e5800200a9a59353d282e7f1eb762961b802810b437bd96fcd5500955: Status 404 returned error can't find the container with id ddbcbf1e5800200a9a59353d282e7f1eb762961b802810b437bd96fcd5500955
Sep 16 18:48:19.813374 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:19.813334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zp7gn" event={"ID":"ec391b92-7e84-445b-9344-ba99b3adc52d","Type":"ContainerStarted","Data":"ddbcbf1e5800200a9a59353d282e7f1eb762961b802810b437bd96fcd5500955"}
Sep 16 18:48:22.827061 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:22.827028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zp7gn" event={"ID":"ec391b92-7e84-445b-9344-ba99b3adc52d","Type":"ContainerStarted","Data":"f233c60bd8573892a2a25ba0730891d9911c1f1d43f5857c6c4578593b6765c3"}
Sep 16 18:48:24.836702 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:24.836669    2579 generic.go:356] "Generic (PLEG): container finished" podID="ec391b92-7e84-445b-9344-ba99b3adc52d" containerID="f233c60bd8573892a2a25ba0730891d9911c1f1d43f5857c6c4578593b6765c3" exitCode=0
Sep 16 18:48:24.837098 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:24.836741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zp7gn" event={"ID":"ec391b92-7e84-445b-9344-ba99b3adc52d","Type":"ContainerDied","Data":"f233c60bd8573892a2a25ba0730891d9911c1f1d43f5857c6c4578593b6765c3"}
Sep 16 18:48:29.864467 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:29.864432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zp7gn" event={"ID":"ec391b92-7e84-445b-9344-ba99b3adc52d","Type":"ContainerStarted","Data":"dece7e6d1f3e7340b43a3726763302062552c879db531eacb1b7c06251dddd6b"}
Sep 16 18:48:29.886884 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:29.886809    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-zp7gn" podStartSLOduration=2.460219089 podStartE2EDuration="11.886792036s" totalImagesPullingTime="7.885129587s" totalInitContainerRuntime="1.54144336s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:29.885646884 +0000 UTC m=+464.550919316" watchObservedRunningTime="2026-09-16 18:48:29.886792036 +0000 UTC m=+464.552064458"
Sep 16 18:48:41.931455 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:41.931416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:42.006070 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:48:42.006036    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:27.577586 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:27.577553    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:35.131723 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:35.131682    2579 generic.go:356] "Generic (PLEG): container finished" podID="ec391b92-7e84-445b-9344-ba99b3adc52d" containerID="dece7e6d1f3e7340b43a3726763302062552c879db531eacb1b7c06251dddd6b" exitCode=0
Sep 16 18:49:35.132058 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:35.131751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zp7gn" event={"ID":"ec391b92-7e84-445b-9344-ba99b3adc52d","Type":"ContainerDied","Data":"dece7e6d1f3e7340b43a3726763302062552c879db531eacb1b7c06251dddd6b"}
Sep 16 18:49:36.261073 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:36.261051    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zp7gn"
Sep 16 18:49:36.364970 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:36.364930    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdmhz\" (UniqueName: \"kubernetes.io/projected/ec391b92-7e84-445b-9344-ba99b3adc52d-kube-api-access-cdmhz\") pod \"ec391b92-7e84-445b-9344-ba99b3adc52d\" (UID: \"ec391b92-7e84-445b-9344-ba99b3adc52d\") "
Sep 16 18:49:36.364970 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:36.364972    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec391b92-7e84-445b-9344-ba99b3adc52d-model-cache\") pod \"ec391b92-7e84-445b-9344-ba99b3adc52d\" (UID: \"ec391b92-7e84-445b-9344-ba99b3adc52d\") "
Sep 16 18:49:36.367389 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:36.367357    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec391b92-7e84-445b-9344-ba99b3adc52d-kube-api-access-cdmhz" (OuterVolumeSpecName: "kube-api-access-cdmhz") pod "ec391b92-7e84-445b-9344-ba99b3adc52d" (UID: "ec391b92-7e84-445b-9344-ba99b3adc52d"). InnerVolumeSpecName "kube-api-access-cdmhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:36.383528 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:36.383499    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec391b92-7e84-445b-9344-ba99b3adc52d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec391b92-7e84-445b-9344-ba99b3adc52d" (UID: "ec391b92-7e84-445b-9344-ba99b3adc52d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:36.466592 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:36.466510    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdmhz\" (UniqueName: \"kubernetes.io/projected/ec391b92-7e84-445b-9344-ba99b3adc52d-kube-api-access-cdmhz\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:49:36.466592 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:36.466539    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec391b92-7e84-445b-9344-ba99b3adc52d-model-cache\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:49:37.140739 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:37.140660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zp7gn" event={"ID":"ec391b92-7e84-445b-9344-ba99b3adc52d","Type":"ContainerDied","Data":"ddbcbf1e5800200a9a59353d282e7f1eb762961b802810b437bd96fcd5500955"}
Sep 16 18:49:37.140739 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:37.140690    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zp7gn"
Sep 16 18:49:37.140739 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:37.140706    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddbcbf1e5800200a9a59353d282e7f1eb762961b802810b437bd96fcd5500955"
Sep 16 18:49:37.504299 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:37.504218    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:37.605681 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:37.605656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.195833 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.195800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.293503 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.293472    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"]
Sep 16 18:49:49.293812 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.293798    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec391b92-7e84-445b-9344-ba99b3adc52d" containerName="download-hf-model"
Sep 16 18:49:49.293859 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.293814    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec391b92-7e84-445b-9344-ba99b3adc52d" containerName="download-hf-model"
Sep 16 18:49:49.293859 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.293821    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec391b92-7e84-445b-9344-ba99b3adc52d" containerName="s3-init"
Sep 16 18:49:49.293859 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.293827    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec391b92-7e84-445b-9344-ba99b3adc52d" containerName="s3-init"
Sep 16 18:49:49.293946 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.293892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec391b92-7e84-445b-9344-ba99b3adc52d" containerName="s3-init"
Sep 16 18:49:49.299543 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.299510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.303339 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.303317    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-35a06-predictor-serving-cert\""
Sep 16 18:49:49.303472 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.303343    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-35a06-kube-rbac-proxy-sar-config\""
Sep 16 18:49:49.303579 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.303565    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-24wmq\""
Sep 16 18:49:49.303646 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.303601    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:49:49.304434 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.304418    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:49:49.316409 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.316388    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"]
Sep 16 18:49:49.367105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.367082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-35a06-predictor-7d699577b8-8szfb\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") " pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.367233 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.367132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29k72\" (UniqueName: \"kubernetes.io/projected/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-api-access-29k72\") pod \"success-200-isvc-35a06-predictor-7d699577b8-8szfb\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") " pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.367233 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.367213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d4f2f9-5842-408a-be24-fd68f9cf6576-proxy-tls\") pod \"success-200-isvc-35a06-predictor-7d699577b8-8szfb\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") " pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.467872 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.467809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29k72\" (UniqueName: \"kubernetes.io/projected/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-api-access-29k72\") pod \"success-200-isvc-35a06-predictor-7d699577b8-8szfb\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") " pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.467971 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.467883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d4f2f9-5842-408a-be24-fd68f9cf6576-proxy-tls\") pod \"success-200-isvc-35a06-predictor-7d699577b8-8szfb\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") " pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.467971 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.467947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-35a06-predictor-7d699577b8-8szfb\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") " pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.468688 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.468666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-35a06-predictor-7d699577b8-8szfb\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") " pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.470541 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.470512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d4f2f9-5842-408a-be24-fd68f9cf6576-proxy-tls\") pod \"success-200-isvc-35a06-predictor-7d699577b8-8szfb\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") " pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.479033 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.479011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29k72\" (UniqueName: \"kubernetes.io/projected/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-api-access-29k72\") pod \"success-200-isvc-35a06-predictor-7d699577b8-8szfb\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") " pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.542529 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.542503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.610268 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.610246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:49:49.623286 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.623263    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"]
Sep 16 18:49:49.627073 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.627052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.629905 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.629885    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Sep 16 18:49:49.630005 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.629982    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:49:49.630263 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.630246    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:49:49.630495 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.630479    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Sep 16 18:49:49.639898 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.639874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.640651 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.640613    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"]
Sep 16 18:49:49.669444 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.669406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.669526 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.669461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9772g\" (UniqueName: \"kubernetes.io/projected/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-api-access-9772g\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.669526 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.669503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.669623 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.669572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec44839-b801-4b2b-abce-fa4298b8a9c7-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.669623 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.669605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.755501 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.755430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.770091 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.770064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec44839-b801-4b2b-abce-fa4298b8a9c7-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.770239 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.770110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.770239 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.770200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.770239 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.770228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9772g\" (UniqueName: \"kubernetes.io/projected/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-api-access-9772g\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.770413 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.770268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.770630 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.770609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.771024 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.771001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.771308 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.771285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.772753 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.772730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec44839-b801-4b2b-abce-fa4298b8a9c7-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.787707 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.787685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9772g\" (UniqueName: \"kubernetes.io/projected/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-api-access-9772g\") pod \"isvc-sklearn-graph-1-predictor-57779f596f-6hgbk\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.854905 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.854882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.895046 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.895023    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"]
Sep 16 18:49:49.899012 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:49:49.898987    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d4f2f9_5842_408a_be24_fd68f9cf6576.slice/crio-2b170e0c33895e209bef14b788db0c0be583a2b6b4e692106979e697f09c285d WatchSource:0}: Error finding container 2b170e0c33895e209bef14b788db0c0be583a2b6b4e692106979e697f09c285d: Status 404 returned error can't find the container with id 2b170e0c33895e209bef14b788db0c0be583a2b6b4e692106979e697f09c285d
Sep 16 18:49:49.921514 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.921490    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"]
Sep 16 18:49:49.926245 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.926228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:49.928993 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.928972    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Sep 16 18:49:49.929461 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.929442    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Sep 16 18:49:49.937128 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.937111    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"]
Sep 16 18:49:49.940860 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.940835    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.943761 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.943742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:49:49.971667 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.971640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8edc3c5-2042-4e56-a44c-e3820b3a0264-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:49.971667 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.971670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj85t\" (UniqueName: \"kubernetes.io/projected/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-api-access-gj85t\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:49.971896 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.971697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:49.971896 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.971741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:49.971896 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:49.971866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.014643 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.014579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:50.032364 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.032338    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"]
Sep 16 18:49:50.036216 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.036192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.038988 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.038970    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Sep 16 18:49:50.039082 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.038975    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Sep 16 18:49:50.046267 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.046246    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"]
Sep 16 18:49:50.073105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.073238 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.073238 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jm68\" (UniqueName: \"kubernetes.io/projected/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-api-access-7jm68\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.073238 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56d9a08-e009-4b38-a53c-36fdec39d00a-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.073372 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8edc3c5-2042-4e56-a44c-e3820b3a0264-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.073372 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj85t\" (UniqueName: \"kubernetes.io/projected/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-api-access-gj85t\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.073372 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56d9a08-e009-4b38-a53c-36fdec39d00a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.073372 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.073531 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.073531 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.073531 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:49:50.073516    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 16 18:49:50.073665 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.073568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.073665 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:49:50.073610    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8edc3c5-2042-4e56-a44c-e3820b3a0264-proxy-tls podName:a8edc3c5-2042-4e56-a44c-e3820b3a0264 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:50.57358624 +0000 UTC m=+545.238858668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a8edc3c5-2042-4e56-a44c-e3820b3a0264-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-jvn9k" (UID: "a8edc3c5-2042-4e56-a44c-e3820b3a0264") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 16 18:49:50.074035 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.074018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.074214 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.074197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.083840 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.083821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj85t\" (UniqueName: \"kubernetes.io/projected/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-api-access-gj85t\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.152321 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.152296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:50.174882 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.174859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.174987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.174897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jm68\" (UniqueName: \"kubernetes.io/projected/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-api-access-7jm68\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.174987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.174929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56d9a08-e009-4b38-a53c-36fdec39d00a-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.175096 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.175057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56d9a08-e009-4b38-a53c-36fdec39d00a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.175179 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.175105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.175264 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:49:50.175243    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 16 18:49:50.175328 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.175311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56d9a08-e009-4b38-a53c-36fdec39d00a-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.175384 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:49:50.175327    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56d9a08-e009-4b38-a53c-36fdec39d00a-proxy-tls podName:e56d9a08-e009-4b38-a53c-36fdec39d00a nodeName:}" failed. No retries permitted until 2026-09-16 18:49:50.675308631 +0000 UTC m=+545.340581054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e56d9a08-e009-4b38-a53c-36fdec39d00a-proxy-tls") pod "isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" (UID: "e56d9a08-e009-4b38-a53c-36fdec39d00a") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 16 18:49:50.175643 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.175620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.175763 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.175744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.185263 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.185240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jm68\" (UniqueName: \"kubernetes.io/projected/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-api-access-7jm68\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.198445 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.198422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" event={"ID":"d5d4f2f9-5842-408a-be24-fd68f9cf6576","Type":"ContainerStarted","Data":"2b170e0c33895e209bef14b788db0c0be583a2b6b4e692106979e697f09c285d"}
Sep 16 18:49:50.222658 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.222640    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"]
Sep 16 18:49:50.224770 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:49:50.224749    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec44839_b801_4b2b_abce_fa4298b8a9c7.slice/crio-58b9c3de65794a1907de75042a7c90117d93727a0a578405b068dfffc5ce15a4 WatchSource:0}: Error finding container 58b9c3de65794a1907de75042a7c90117d93727a0a578405b068dfffc5ce15a4: Status 404 returned error can't find the container with id 58b9c3de65794a1907de75042a7c90117d93727a0a578405b068dfffc5ce15a4
Sep 16 18:49:50.344958 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.344922    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:50.578447 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.578411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8edc3c5-2042-4e56-a44c-e3820b3a0264-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.581921 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.581871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8edc3c5-2042-4e56-a44c-e3820b3a0264-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jvn9k\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.680343 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.679833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56d9a08-e009-4b38-a53c-36fdec39d00a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.688812 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.688739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56d9a08-e009-4b38-a53c-36fdec39d00a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6b4888bd86-vt525\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:50.807284 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.807109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:50.837079 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.836862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:49:50.947518 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:50.947482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:49:51.168929 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:51.168646    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"]
Sep 16 18:49:51.173652 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:49:51.173621    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8edc3c5_2042_4e56_a44c_e3820b3a0264.slice/crio-c8f4da891d32dc50bf6afbc7e31242d0346bcf11d56210af405692d1a7e2a1cb WatchSource:0}: Error finding container c8f4da891d32dc50bf6afbc7e31242d0346bcf11d56210af405692d1a7e2a1cb: Status 404 returned error can't find the container with id c8f4da891d32dc50bf6afbc7e31242d0346bcf11d56210af405692d1a7e2a1cb
Sep 16 18:49:51.212894 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:51.212860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" event={"ID":"3ec44839-b801-4b2b-abce-fa4298b8a9c7","Type":"ContainerStarted","Data":"7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd"}
Sep 16 18:49:51.213234 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:51.212938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" event={"ID":"3ec44839-b801-4b2b-abce-fa4298b8a9c7","Type":"ContainerStarted","Data":"58b9c3de65794a1907de75042a7c90117d93727a0a578405b068dfffc5ce15a4"}
Sep 16 18:49:51.216217 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:51.216175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" event={"ID":"a8edc3c5-2042-4e56-a44c-e3820b3a0264","Type":"ContainerStarted","Data":"c8f4da891d32dc50bf6afbc7e31242d0346bcf11d56210af405692d1a7e2a1cb"}
Sep 16 18:49:51.265797 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:51.265566    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"]
Sep 16 18:49:51.269101 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:49:51.269072    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56d9a08_e009_4b38_a53c_36fdec39d00a.slice/crio-ae04afdb164d5af465f8475b2ef9de3d20f6edd2137fbaa15add7eb62ba54158 WatchSource:0}: Error finding container ae04afdb164d5af465f8475b2ef9de3d20f6edd2137fbaa15add7eb62ba54158: Status 404 returned error can't find the container with id ae04afdb164d5af465f8475b2ef9de3d20f6edd2137fbaa15add7eb62ba54158
Sep 16 18:49:52.229871 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:52.228918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" event={"ID":"e56d9a08-e009-4b38-a53c-36fdec39d00a","Type":"ContainerStarted","Data":"1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e"}
Sep 16 18:49:52.229871 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:52.229033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" event={"ID":"e56d9a08-e009-4b38-a53c-36fdec39d00a","Type":"ContainerStarted","Data":"ae04afdb164d5af465f8475b2ef9de3d20f6edd2137fbaa15add7eb62ba54158"}
Sep 16 18:49:52.234736 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:52.233514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" event={"ID":"a8edc3c5-2042-4e56-a44c-e3820b3a0264","Type":"ContainerStarted","Data":"ab81706ffd2b6d33efd4e2430bcd13533e002225aa40694da3e8cf84dd72a6e4"}
Sep 16 18:49:55.256588 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:55.256502    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerID="7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd" exitCode=0
Sep 16 18:49:55.256588 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:55.256581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" event={"ID":"3ec44839-b801-4b2b-abce-fa4298b8a9c7","Type":"ContainerDied","Data":"7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd"}
Sep 16 18:49:59.282126 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:59.282040    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerID="ab81706ffd2b6d33efd4e2430bcd13533e002225aa40694da3e8cf84dd72a6e4" exitCode=0
Sep 16 18:49:59.282595 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:59.282243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" event={"ID":"a8edc3c5-2042-4e56-a44c-e3820b3a0264","Type":"ContainerDied","Data":"ab81706ffd2b6d33efd4e2430bcd13533e002225aa40694da3e8cf84dd72a6e4"}
Sep 16 18:49:59.285875 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:59.285843    2579 generic.go:356] "Generic (PLEG): container finished" podID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerID="1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e" exitCode=0
Sep 16 18:49:59.285988 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:49:59.285901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" event={"ID":"e56d9a08-e009-4b38-a53c-36fdec39d00a","Type":"ContainerDied","Data":"1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e"}
Sep 16 18:50:03.316932 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:03.316894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" event={"ID":"d5d4f2f9-5842-408a-be24-fd68f9cf6576","Type":"ContainerStarted","Data":"340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1"}
Sep 16 18:50:12.387486 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:12.387403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" event={"ID":"3ec44839-b801-4b2b-abce-fa4298b8a9c7","Type":"ContainerStarted","Data":"5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95"}
Sep 16 18:50:12.389870 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:12.389843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" event={"ID":"e56d9a08-e009-4b38-a53c-36fdec39d00a","Type":"ContainerStarted","Data":"d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6"}
Sep 16 18:50:12.392250 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:12.392220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" event={"ID":"d5d4f2f9-5842-408a-be24-fd68f9cf6576","Type":"ContainerStarted","Data":"f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae"}
Sep 16 18:50:12.392454 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:12.392432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:50:12.392527 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:12.392493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:50:12.393602 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:12.393571    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:50:12.417425 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:12.417373    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" podStartSLOduration=1.307835686 podStartE2EDuration="23.41735342s" totalImagesPullingTime="22.109517734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:12.415903751 +0000 UTC m=+567.081176202" watchObservedRunningTime="2026-09-16 18:50:12.41735342 +0000 UTC m=+567.082625856"
Sep 16 18:50:13.400449 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:13.400416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" event={"ID":"3ec44839-b801-4b2b-abce-fa4298b8a9c7","Type":"ContainerStarted","Data":"efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114"}
Sep 16 18:50:13.400871 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:13.400553    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:50:13.402871 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:13.402837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" event={"ID":"e56d9a08-e009-4b38-a53c-36fdec39d00a","Type":"ContainerStarted","Data":"5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c"}
Sep 16 18:50:13.403085 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:13.403068    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:50:13.403181 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:13.403119    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:50:13.403249 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:13.403212    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:50:13.403989 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:13.403963    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 18:50:13.424015 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:13.423967    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podStartSLOduration=3.956358089 podStartE2EDuration="24.423948293s" totalImagesPullingTime="16.475953052s" totalInitContainerRuntime="3.991637152s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:13.421389602 +0000 UTC m=+568.086662059" watchObservedRunningTime="2026-09-16 18:50:13.423948293 +0000 UTC m=+568.089220724"
Sep 16 18:50:13.443622 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:13.443568    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podStartSLOduration=6.326957735 podStartE2EDuration="23.443553994s" totalImagesPullingTime="12.81243454s" totalInitContainerRuntime="4.304161719s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:13.441274885 +0000 UTC m=+568.106547334" watchObservedRunningTime="2026-09-16 18:50:13.443553994 +0000 UTC m=+568.108826423"
Sep 16 18:50:14.410484 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:14.410431    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 18:50:14.410951 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:14.410694    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:50:14.411971 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:14.411942    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:50:15.414930 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:15.414887    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:50:18.408241 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:18.408208    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:50:18.408717 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:18.408684    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:50:19.416466 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:19.416437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:50:19.417023 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:19.416986    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 18:50:20.422233 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:20.422200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:50:20.422894 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:20.422860    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:50:22.454120 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:22.454070    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" event={"ID":"a8edc3c5-2042-4e56-a44c-e3820b3a0264","Type":"ContainerStarted","Data":"d14f290b8916a77b314886f8ddc9b7e960e52d6cf3788c19f579ba8b1a67c378"}
Sep 16 18:50:22.454593 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:22.454171    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" event={"ID":"a8edc3c5-2042-4e56-a44c-e3820b3a0264","Type":"ContainerStarted","Data":"9706edb4e5f61530c6e9c3aeb36739cde64a223536e96fd8c9c5cfc289f991e0"}
Sep 16 18:50:22.454593 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:22.454417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:50:22.481751 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:22.481699    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podStartSLOduration=6.409786241 podStartE2EDuration="33.481680016s" totalImagesPullingTime="22.713723216s" totalInitContainerRuntime="4.358170559s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:22.480780262 +0000 UTC m=+577.146052711" watchObservedRunningTime="2026-09-16 18:50:22.481680016 +0000 UTC m=+577.146952451"
Sep 16 18:50:23.461728 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:23.461692    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:50:23.463394 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:23.463365    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:50:24.466454 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:24.466416    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:50:28.409275 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:28.409237    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:50:29.417189 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:29.417124    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 18:50:29.470870 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:29.470842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:50:29.471404 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:29.471376    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:50:30.423603 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:30.423559    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:50:31.290641 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:31.290606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:31.390233 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:31.390145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:38.409081 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:38.409038    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:50:39.417336 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:39.417301    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 18:50:39.472290 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:39.472252    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:50:40.423418 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:40.423380    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:50:47.632109 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:47.632075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:48.409400 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:48.409356    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:50:49.416930 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:49.416892    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 18:50:49.471353 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:49.471321    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:50:50.423039 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:50.423003    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:50:58.410203 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:58.410174    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:50:59.417237 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:59.417195    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 18:50:59.472103 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:50:59.472070    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:51:00.423245 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:00.423209    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:51:09.315767 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:09.315730    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:09.416900 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:09.416862    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 18:51:09.471389 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:09.471346    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:51:09.544277 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:09.544238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:10.423541 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:10.423502    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:51:19.417648 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.417607    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 18:51:19.471323 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.471290    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:51:19.544983 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.544955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:19.606338 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.606304    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"]
Sep 16 18:51:19.606639 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.606586    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kserve-container" containerID="cri-o://340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1" gracePeriod=30
Sep 16 18:51:19.606790 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.606656    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kube-rbac-proxy" containerID="cri-o://f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae" gracePeriod=30
Sep 16 18:51:19.727105 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.727061    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerID="f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae" exitCode=2
Sep 16 18:51:19.727266 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.727140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" event={"ID":"d5d4f2f9-5842-408a-be24-fd68f9cf6576","Type":"ContainerDied","Data":"f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae"}
Sep 16 18:51:19.731808 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.731783    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"]
Sep 16 18:51:19.734382 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.734362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:19.736953 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.736932    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-15f71-kube-rbac-proxy-sar-config\""
Sep 16 18:51:19.736953 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.736943    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-15f71-predictor-serving-cert\""
Sep 16 18:51:19.745674 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.745655    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"]
Sep 16 18:51:19.805740 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.805712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-15f71-predictor-784478884b-fqkdn\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") " pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:19.805903 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.805789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwb2g\" (UniqueName: \"kubernetes.io/projected/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-api-access-vwb2g\") pod \"success-200-isvc-15f71-predictor-784478884b-fqkdn\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") " pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:19.805903 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.805840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df06d1f7-4675-4c60-a877-e3c3ac93c475-proxy-tls\") pod \"success-200-isvc-15f71-predictor-784478884b-fqkdn\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") " pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:19.906721 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.906685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-15f71-predictor-784478884b-fqkdn\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") " pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:19.906884 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.906778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwb2g\" (UniqueName: \"kubernetes.io/projected/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-api-access-vwb2g\") pod \"success-200-isvc-15f71-predictor-784478884b-fqkdn\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") " pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:19.906884 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.906834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df06d1f7-4675-4c60-a877-e3c3ac93c475-proxy-tls\") pod \"success-200-isvc-15f71-predictor-784478884b-fqkdn\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") " pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:19.907005 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:51:19.906988    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-15f71-predictor-serving-cert: secret "success-200-isvc-15f71-predictor-serving-cert" not found
Sep 16 18:51:19.907068 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:51:19.907058    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df06d1f7-4675-4c60-a877-e3c3ac93c475-proxy-tls podName:df06d1f7-4675-4c60-a877-e3c3ac93c475 nodeName:}" failed. No retries permitted until 2026-09-16 18:51:20.407038099 +0000 UTC m=+635.072310526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/df06d1f7-4675-4c60-a877-e3c3ac93c475-proxy-tls") pod "success-200-isvc-15f71-predictor-784478884b-fqkdn" (UID: "df06d1f7-4675-4c60-a877-e3c3ac93c475") : secret "success-200-isvc-15f71-predictor-serving-cert" not found
Sep 16 18:51:19.907415 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.907400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-15f71-predictor-784478884b-fqkdn\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") " pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:19.917166 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:19.917120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwb2g\" (UniqueName: \"kubernetes.io/projected/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-api-access-vwb2g\") pod \"success-200-isvc-15f71-predictor-784478884b-fqkdn\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") " pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:20.411138 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:20.411101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df06d1f7-4675-4c60-a877-e3c3ac93c475-proxy-tls\") pod \"success-200-isvc-15f71-predictor-784478884b-fqkdn\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") " pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:20.413577 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:20.413549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df06d1f7-4675-4c60-a877-e3c3ac93c475-proxy-tls\") pod \"success-200-isvc-15f71-predictor-784478884b-fqkdn\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") " pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:20.423215 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:20.423184    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:51:20.647754 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:20.647722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:20.764094 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:20.764066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:20.922575 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:20.922545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"]
Sep 16 18:51:20.924463 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:51:20.924439    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf06d1f7_4675_4c60_a877_e3c3ac93c475.slice/crio-e0f76e8767d7e13385147a529af6e880e9e50af547abdf2a9544d3ad7440f2bb WatchSource:0}: Error finding container e0f76e8767d7e13385147a529af6e880e9e50af547abdf2a9544d3ad7440f2bb: Status 404 returned error can't find the container with id e0f76e8767d7e13385147a529af6e880e9e50af547abdf2a9544d3ad7440f2bb
Sep 16 18:51:21.738390 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:21.738352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" event={"ID":"df06d1f7-4675-4c60-a877-e3c3ac93c475","Type":"ContainerStarted","Data":"74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056"}
Sep 16 18:51:21.738815 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:21.738420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" event={"ID":"df06d1f7-4675-4c60-a877-e3c3ac93c475","Type":"ContainerStarted","Data":"b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13"}
Sep 16 18:51:21.738815 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:21.738459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" event={"ID":"df06d1f7-4675-4c60-a877-e3c3ac93c475","Type":"ContainerStarted","Data":"e0f76e8767d7e13385147a529af6e880e9e50af547abdf2a9544d3ad7440f2bb"}
Sep 16 18:51:21.738815 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:21.738571    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:21.738815 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:21.738624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:21.739783 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:21.739731    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:21.759335 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:21.759285    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podStartSLOduration=2.759270121 podStartE2EDuration="2.759270121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:21.758249564 +0000 UTC m=+636.423522018" watchObservedRunningTime="2026-09-16 18:51:21.759270121 +0000 UTC m=+636.424542551"
Sep 16 18:51:22.742973 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:22.742937    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:23.152439 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.152413    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:51:23.236609 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.236582    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29k72\" (UniqueName: \"kubernetes.io/projected/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-api-access-29k72\") pod \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") "
Sep 16 18:51:23.236724 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.236621    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d4f2f9-5842-408a-be24-fd68f9cf6576-proxy-tls\") pod \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") "
Sep 16 18:51:23.236724 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.236659    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-rbac-proxy-sar-config\") pod \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\" (UID: \"d5d4f2f9-5842-408a-be24-fd68f9cf6576\") "
Sep 16 18:51:23.243956 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.237460    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5d4f2f9-5842-408a-be24-fd68f9cf6576" (UID: "d5d4f2f9-5842-408a-be24-fd68f9cf6576"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:23.243956 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.239602    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d4f2f9-5842-408a-be24-fd68f9cf6576-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5d4f2f9-5842-408a-be24-fd68f9cf6576" (UID: "d5d4f2f9-5842-408a-be24-fd68f9cf6576"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:23.243956 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.239675    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-api-access-29k72" (OuterVolumeSpecName: "kube-api-access-29k72") pod "d5d4f2f9-5842-408a-be24-fd68f9cf6576" (UID: "d5d4f2f9-5842-408a-be24-fd68f9cf6576"). InnerVolumeSpecName "kube-api-access-29k72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:23.337301 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.337273    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d4f2f9-5842-408a-be24-fd68f9cf6576-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:51:23.337301 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.337296    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:51:23.337456 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.337307    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29k72\" (UniqueName: \"kubernetes.io/projected/d5d4f2f9-5842-408a-be24-fd68f9cf6576-kube-api-access-29k72\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:51:23.750453 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.750357    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerID="340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1" exitCode=0
Sep 16 18:51:23.750453 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.750427    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"
Sep 16 18:51:23.750951 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.750447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" event={"ID":"d5d4f2f9-5842-408a-be24-fd68f9cf6576","Type":"ContainerDied","Data":"340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1"}
Sep 16 18:51:23.750951 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.750531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb" event={"ID":"d5d4f2f9-5842-408a-be24-fd68f9cf6576","Type":"ContainerDied","Data":"2b170e0c33895e209bef14b788db0c0be583a2b6b4e692106979e697f09c285d"}
Sep 16 18:51:23.750951 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.750561    2579 scope.go:117] "RemoveContainer" containerID="f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae"
Sep 16 18:51:23.759568 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.759479    2579 scope.go:117] "RemoveContainer" containerID="340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1"
Sep 16 18:51:23.767751 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.767733    2579 scope.go:117] "RemoveContainer" containerID="f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae"
Sep 16 18:51:23.768010 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:51:23.767990    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae\": container with ID starting with f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae not found: ID does not exist" containerID="f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae"
Sep 16 18:51:23.768085 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.768019    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae"} err="failed to get container status \"f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae\": rpc error: code = NotFound desc = could not find container \"f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae\": container with ID starting with f4cf6597f872417d5de5023cbad45bd856e0d163e48d30d7320278a1f087ddae not found: ID does not exist"
Sep 16 18:51:23.768085 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.768036    2579 scope.go:117] "RemoveContainer" containerID="340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1"
Sep 16 18:51:23.768307 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:51:23.768290    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1\": container with ID starting with 340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1 not found: ID does not exist" containerID="340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1"
Sep 16 18:51:23.768360 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.768313    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1"} err="failed to get container status \"340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1\": rpc error: code = NotFound desc = could not find container \"340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1\": container with ID starting with 340f2ff98a8298147cbb0397ea2882f6bc21fd07aee48b28a17f63b27f7725a1 not found: ID does not exist"
Sep 16 18:51:23.780826 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.780803    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"]
Sep 16 18:51:23.790898 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.790874    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35a06-predictor-7d699577b8-8szfb"]
Sep 16 18:51:23.811319 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:23.811294    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" path="/var/lib/kubelet/pods/d5d4f2f9-5842-408a-be24-fd68f9cf6576/volumes"
Sep 16 18:51:27.747681 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:27.747653    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:51:27.748070 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:27.748046    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:29.417024 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:29.416982    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 18:51:29.472831 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:29.472803    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:51:30.423899 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:30.423867    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:51:37.748305 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:37.748260    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:38.807316 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:38.807278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:51:47.749005 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:47.748964    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:57.748541 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:57.748501    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:59.322032 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:59.322001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:59.689562 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:51:59.689456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:07.748692 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:07.748666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 18:52:09.383423 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.383393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:09.451694 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.451652    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"]
Sep 16 18:52:09.451987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.451947    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kube-rbac-proxy" containerID="cri-o://5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c" gracePeriod=30
Sep 16 18:52:09.452138 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.451935    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container" containerID="cri-o://d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6" gracePeriod=30
Sep 16 18:52:09.480772 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.480746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:09.488731 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.488700    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"]
Sep 16 18:52:09.488955 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.488926    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" containerID="cri-o://5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95" gracePeriod=30
Sep 16 18:52:09.489097 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.488955    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kube-rbac-proxy" containerID="cri-o://efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114" gracePeriod=30
Sep 16 18:52:09.525518 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.525491    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"]
Sep 16 18:52:09.525746 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.525725    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container" containerID="cri-o://9706edb4e5f61530c6e9c3aeb36739cde64a223536e96fd8c9c5cfc289f991e0" gracePeriod=30
Sep 16 18:52:09.525847 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.525817    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kube-rbac-proxy" containerID="cri-o://d14f290b8916a77b314886f8ddc9b7e960e52d6cf3788c19f579ba8b1a67c378" gracePeriod=30
Sep 16 18:52:09.561609 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.561577    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:09.614191 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.614146    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"]
Sep 16 18:52:09.614574 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.614560    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kube-rbac-proxy"
Sep 16 18:52:09.614621 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.614575    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kube-rbac-proxy"
Sep 16 18:52:09.614621 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.614590    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kserve-container"
Sep 16 18:52:09.614621 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.614596    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kserve-container"
Sep 16 18:52:09.614713 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.614655    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kube-rbac-proxy"
Sep 16 18:52:09.614713 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.614667    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d4f2f9-5842-408a-be24-fd68f9cf6576" containerName="kserve-container"
Sep 16 18:52:09.616624 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.616609    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.619412 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.619393    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-21301-kube-rbac-proxy-sar-config\""
Sep 16 18:52:09.619524 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.619482    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-21301-predictor-serving-cert\""
Sep 16 18:52:09.634292 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.634244    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"]
Sep 16 18:52:09.638065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.638042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21301-predictor-68796d9f44-xxrjr\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") " pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.638181 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.638072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9m5\" (UniqueName: \"kubernetes.io/projected/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-api-access-zn9m5\") pod \"success-200-isvc-21301-predictor-68796d9f44-xxrjr\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") " pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.638237 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.638190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/242b5f6f-7683-4f38-8ae8-1e17a985de8d-proxy-tls\") pod \"success-200-isvc-21301-predictor-68796d9f44-xxrjr\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") " pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.679608 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.679577    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:09.739379 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.739348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/242b5f6f-7683-4f38-8ae8-1e17a985de8d-proxy-tls\") pod \"success-200-isvc-21301-predictor-68796d9f44-xxrjr\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") " pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.739530 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.739429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21301-predictor-68796d9f44-xxrjr\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") " pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.739530 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.739450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9m5\" (UniqueName: \"kubernetes.io/projected/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-api-access-zn9m5\") pod \"success-200-isvc-21301-predictor-68796d9f44-xxrjr\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") " pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.740346 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.740320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21301-predictor-68796d9f44-xxrjr\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") " pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.741876 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.741859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/242b5f6f-7683-4f38-8ae8-1e17a985de8d-proxy-tls\") pod \"success-200-isvc-21301-predictor-68796d9f44-xxrjr\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") " pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.755054 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.755003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9m5\" (UniqueName: \"kubernetes.io/projected/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-api-access-zn9m5\") pod \"success-200-isvc-21301-predictor-68796d9f44-xxrjr\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") " pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.780822 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.780789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:09.866375 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.866339    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:09.926926 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.926836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:09.973969 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.973931    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerID="d14f290b8916a77b314886f8ddc9b7e960e52d6cf3788c19f579ba8b1a67c378" exitCode=2
Sep 16 18:52:09.974131 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.973982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" event={"ID":"a8edc3c5-2042-4e56-a44c-e3820b3a0264","Type":"ContainerDied","Data":"d14f290b8916a77b314886f8ddc9b7e960e52d6cf3788c19f579ba8b1a67c378"}
Sep 16 18:52:09.976411 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.976391    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerID="efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114" exitCode=2
Sep 16 18:52:09.976547 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.976461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" event={"ID":"3ec44839-b801-4b2b-abce-fa4298b8a9c7","Type":"ContainerDied","Data":"efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114"}
Sep 16 18:52:09.978713 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.978692    2579 generic.go:356] "Generic (PLEG): container finished" podID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerID="5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c" exitCode=2
Sep 16 18:52:09.978801 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.978736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" event={"ID":"e56d9a08-e009-4b38-a53c-36fdec39d00a","Type":"ContainerDied","Data":"5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c"}
Sep 16 18:52:09.999788 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:09.999757    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:10.210050 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:10.210023    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"]
Sep 16 18:52:10.211823 ip-10-0-140-67 kubenswrapper[2579]: W0916 18:52:10.211802    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242b5f6f_7683_4f38_8ae8_1e17a985de8d.slice/crio-cc7b50e846cf8585a0cd626dc905230abe0280ce992aaa166b22516de3c9c908 WatchSource:0}: Error finding container cc7b50e846cf8585a0cd626dc905230abe0280ce992aaa166b22516de3c9c908: Status 404 returned error can't find the container with id cc7b50e846cf8585a0cd626dc905230abe0280ce992aaa166b22516de3c9c908
Sep 16 18:52:10.415406 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:10.415363    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 16 18:52:10.423134 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:10.423103    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:52:10.985208 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:10.985143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" event={"ID":"242b5f6f-7683-4f38-8ae8-1e17a985de8d","Type":"ContainerStarted","Data":"b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62"}
Sep 16 18:52:10.985417 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:10.985237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:10.985417 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:10.985258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" event={"ID":"242b5f6f-7683-4f38-8ae8-1e17a985de8d","Type":"ContainerStarted","Data":"09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e"}
Sep 16 18:52:10.985417 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:10.985276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" event={"ID":"242b5f6f-7683-4f38-8ae8-1e17a985de8d","Type":"ContainerStarted","Data":"cc7b50e846cf8585a0cd626dc905230abe0280ce992aaa166b22516de3c9c908"}
Sep 16 18:52:10.985417 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:10.985297    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:10.986740 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:10.986717    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:52:11.009017 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:11.008970    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podStartSLOduration=2.008949521 podStartE2EDuration="2.008949521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:11.006943239 +0000 UTC m=+685.672215705" watchObservedRunningTime="2026-09-16 18:52:11.008949521 +0000 UTC m=+685.674221950"
Sep 16 18:52:11.990506 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:11.990469    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:52:14.004065 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.004037    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerID="9706edb4e5f61530c6e9c3aeb36739cde64a223536e96fd8c9c5cfc289f991e0" exitCode=0
Sep 16 18:52:14.004415 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.004094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" event={"ID":"a8edc3c5-2042-4e56-a44c-e3820b3a0264","Type":"ContainerDied","Data":"9706edb4e5f61530c6e9c3aeb36739cde64a223536e96fd8c9c5cfc289f991e0"}
Sep 16 18:52:14.072753 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.072731    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:52:14.179874 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.179841    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-rbac-proxy-sar-config\") pod \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") "
Sep 16 18:52:14.179991 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.179909    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kserve-provision-location\") pod \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") "
Sep 16 18:52:14.179991 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.179942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8edc3c5-2042-4e56-a44c-e3820b3a0264-proxy-tls\") pod \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") "
Sep 16 18:52:14.179991 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.179961    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj85t\" (UniqueName: \"kubernetes.io/projected/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-api-access-gj85t\") pod \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") "
Sep 16 18:52:14.180128 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.180017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-cabundle-cert\") pod \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\" (UID: \"a8edc3c5-2042-4e56-a44c-e3820b3a0264\") "
Sep 16 18:52:14.180490 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.180462    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8edc3c5-2042-4e56-a44c-e3820b3a0264" (UID: "a8edc3c5-2042-4e56-a44c-e3820b3a0264"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:14.180565 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.180501    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8edc3c5-2042-4e56-a44c-e3820b3a0264" (UID: "a8edc3c5-2042-4e56-a44c-e3820b3a0264"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:14.180565 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.180506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8edc3c5-2042-4e56-a44c-e3820b3a0264" (UID: "a8edc3c5-2042-4e56-a44c-e3820b3a0264"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:14.182222 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.182194    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8edc3c5-2042-4e56-a44c-e3820b3a0264-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8edc3c5-2042-4e56-a44c-e3820b3a0264" (UID: "a8edc3c5-2042-4e56-a44c-e3820b3a0264"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:14.182300 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.182285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-api-access-gj85t" (OuterVolumeSpecName: "kube-api-access-gj85t") pod "a8edc3c5-2042-4e56-a44c-e3820b3a0264" (UID: "a8edc3c5-2042-4e56-a44c-e3820b3a0264"). InnerVolumeSpecName "kube-api-access-gj85t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:14.281039 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.281008    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kserve-provision-location\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.281197 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.281042    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8edc3c5-2042-4e56-a44c-e3820b3a0264-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.281197 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.281059    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj85t\" (UniqueName: \"kubernetes.io/projected/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-api-access-gj85t\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.281197 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.281071    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-cabundle-cert\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.281197 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.281084    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8edc3c5-2042-4e56-a44c-e3820b3a0264-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.365209 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.365187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:52:14.411442 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.411402    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 16 18:52:14.482684 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.482660    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec44839-b801-4b2b-abce-fa4298b8a9c7-proxy-tls\") pod \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") "
Sep 16 18:52:14.482775 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.482758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-rbac-proxy-sar-config\") pod \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") "
Sep 16 18:52:14.482823 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.482812    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-cabundle-cert\") pod \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") "
Sep 16 18:52:14.482864 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.482841    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kserve-provision-location\") pod \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") "
Sep 16 18:52:14.482910 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.482876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9772g\" (UniqueName: \"kubernetes.io/projected/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-api-access-9772g\") pod \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\" (UID: \"3ec44839-b801-4b2b-abce-fa4298b8a9c7\") "
Sep 16 18:52:14.483355 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.483145    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ec44839-b801-4b2b-abce-fa4298b8a9c7" (UID: "3ec44839-b801-4b2b-abce-fa4298b8a9c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:14.483355 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.483168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ec44839-b801-4b2b-abce-fa4298b8a9c7" (UID: "3ec44839-b801-4b2b-abce-fa4298b8a9c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:14.483355 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.483208    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ec44839-b801-4b2b-abce-fa4298b8a9c7" (UID: "3ec44839-b801-4b2b-abce-fa4298b8a9c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:14.484752 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.484726    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec44839-b801-4b2b-abce-fa4298b8a9c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ec44839-b801-4b2b-abce-fa4298b8a9c7" (UID: "3ec44839-b801-4b2b-abce-fa4298b8a9c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:14.485261 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.485231    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-api-access-9772g" (OuterVolumeSpecName: "kube-api-access-9772g") pod "3ec44839-b801-4b2b-abce-fa4298b8a9c7" (UID: "3ec44839-b801-4b2b-abce-fa4298b8a9c7"). InnerVolumeSpecName "kube-api-access-9772g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:14.490526 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.490496    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:52:14.584077 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584050    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56d9a08-e009-4b38-a53c-36fdec39d00a-proxy-tls\") pod \"e56d9a08-e009-4b38-a53c-36fdec39d00a\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") "
Sep 16 18:52:14.584213 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584094    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jm68\" (UniqueName: \"kubernetes.io/projected/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-api-access-7jm68\") pod \"e56d9a08-e009-4b38-a53c-36fdec39d00a\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") "
Sep 16 18:52:14.584213 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56d9a08-e009-4b38-a53c-36fdec39d00a-kserve-provision-location\") pod \"e56d9a08-e009-4b38-a53c-36fdec39d00a\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") "
Sep 16 18:52:14.584302 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-rbac-proxy-sar-config\") pod \"e56d9a08-e009-4b38-a53c-36fdec39d00a\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") "
Sep 16 18:52:14.584302 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584294    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-cabundle-cert\") pod \"e56d9a08-e009-4b38-a53c-36fdec39d00a\" (UID: \"e56d9a08-e009-4b38-a53c-36fdec39d00a\") "
Sep 16 18:52:14.584567 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56d9a08-e009-4b38-a53c-36fdec39d00a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e56d9a08-e009-4b38-a53c-36fdec39d00a" (UID: "e56d9a08-e009-4b38-a53c-36fdec39d00a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:14.584670 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584542    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.584670 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584664    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ec44839-b801-4b2b-abce-fa4298b8a9c7-cabundle-cert\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.584755 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584669    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e56d9a08-e009-4b38-a53c-36fdec39d00a" (UID: "e56d9a08-e009-4b38-a53c-36fdec39d00a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:14.584755 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584686    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kserve-provision-location\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.584755 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584703    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9772g\" (UniqueName: \"kubernetes.io/projected/3ec44839-b801-4b2b-abce-fa4298b8a9c7-kube-api-access-9772g\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.584755 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584707    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e56d9a08-e009-4b38-a53c-36fdec39d00a" (UID: "e56d9a08-e009-4b38-a53c-36fdec39d00a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:14.584755 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.584720    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec44839-b801-4b2b-abce-fa4298b8a9c7-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.586258 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.586238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56d9a08-e009-4b38-a53c-36fdec39d00a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e56d9a08-e009-4b38-a53c-36fdec39d00a" (UID: "e56d9a08-e009-4b38-a53c-36fdec39d00a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:14.586341 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.586308    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-api-access-7jm68" (OuterVolumeSpecName: "kube-api-access-7jm68") pod "e56d9a08-e009-4b38-a53c-36fdec39d00a" (UID: "e56d9a08-e009-4b38-a53c-36fdec39d00a"). InnerVolumeSpecName "kube-api-access-7jm68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:14.686113 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.686045    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jm68\" (UniqueName: \"kubernetes.io/projected/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-api-access-7jm68\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.686113 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.686080    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56d9a08-e009-4b38-a53c-36fdec39d00a-kserve-provision-location\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.686113 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.686095    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.686113 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.686107    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e56d9a08-e009-4b38-a53c-36fdec39d00a-cabundle-cert\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:14.686324 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:14.686122    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56d9a08-e009-4b38-a53c-36fdec39d00a-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 18:52:15.011499 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.011426    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerID="5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95" exitCode=0
Sep 16 18:52:15.011499 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.011476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" event={"ID":"3ec44839-b801-4b2b-abce-fa4298b8a9c7","Type":"ContainerDied","Data":"5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95"}
Sep 16 18:52:15.011499 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.011492    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"
Sep 16 18:52:15.011980 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.011543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk" event={"ID":"3ec44839-b801-4b2b-abce-fa4298b8a9c7","Type":"ContainerDied","Data":"58b9c3de65794a1907de75042a7c90117d93727a0a578405b068dfffc5ce15a4"}
Sep 16 18:52:15.011980 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.011585    2579 scope.go:117] "RemoveContainer" containerID="efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114"
Sep 16 18:52:15.013728 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.013704    2579 generic.go:356] "Generic (PLEG): container finished" podID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerID="d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6" exitCode=0
Sep 16 18:52:15.013841 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.013765    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"
Sep 16 18:52:15.013841 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.013787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" event={"ID":"e56d9a08-e009-4b38-a53c-36fdec39d00a","Type":"ContainerDied","Data":"d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6"}
Sep 16 18:52:15.013931 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.013843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525" event={"ID":"e56d9a08-e009-4b38-a53c-36fdec39d00a","Type":"ContainerDied","Data":"ae04afdb164d5af465f8475b2ef9de3d20f6edd2137fbaa15add7eb62ba54158"}
Sep 16 18:52:15.016260 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.016239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k" event={"ID":"a8edc3c5-2042-4e56-a44c-e3820b3a0264","Type":"ContainerDied","Data":"c8f4da891d32dc50bf6afbc7e31242d0346bcf11d56210af405692d1a7e2a1cb"}
Sep 16 18:52:15.016365 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.016242    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"
Sep 16 18:52:15.021958 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.021942    2579 scope.go:117] "RemoveContainer" containerID="5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95"
Sep 16 18:52:15.037956 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.037933    2579 scope.go:117] "RemoveContainer" containerID="7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd"
Sep 16 18:52:15.045533 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.045514    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"]
Sep 16 18:52:15.045784 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.045771    2579 scope.go:117] "RemoveContainer" containerID="efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114"
Sep 16 18:52:15.046077 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:52:15.046058    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114\": container with ID starting with efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114 not found: ID does not exist" containerID="efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114"
Sep 16 18:52:15.046113 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.046086    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114"} err="failed to get container status \"efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114\": rpc error: code = NotFound desc = could not find container \"efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114\": container with ID starting with efffb9ba8d1f00ddf6524c356a090d3a960b622f6fa1bc80cdba987f57254114 not found: ID does not exist"
Sep 16 18:52:15.046113 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.046103    2579 scope.go:117] "RemoveContainer" containerID="5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95"
Sep 16 18:52:15.046366 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:52:15.046350    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95\": container with ID starting with 5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95 not found: ID does not exist" containerID="5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95"
Sep 16 18:52:15.046399 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.046373    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95"} err="failed to get container status \"5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95\": rpc error: code = NotFound desc = could not find container \"5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95\": container with ID starting with 5d410c9c8cf4e362da28cef822a22a1b6ccb34c5e568a3d61b4fbc70ecc4ee95 not found: ID does not exist"
Sep 16 18:52:15.046399 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.046387    2579 scope.go:117] "RemoveContainer" containerID="7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd"
Sep 16 18:52:15.046617 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:52:15.046596    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd\": container with ID starting with 7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd not found: ID does not exist" containerID="7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd"
Sep 16 18:52:15.046695 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.046625    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd"} err="failed to get container status \"7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd\": rpc error: code = NotFound desc = could not find container \"7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd\": container with ID starting with 7492b9f6a30b8e8d00af6efe8adaa48054ec0406d4714d1c7ea19a2edb6af2bd not found: ID does not exist"
Sep 16 18:52:15.046695 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.046646    2579 scope.go:117] "RemoveContainer" containerID="5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c"
Sep 16 18:52:15.050218 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.050189    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57779f596f-6hgbk"]
Sep 16 18:52:15.054620 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.054594    2579 scope.go:117] "RemoveContainer" containerID="d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6"
Sep 16 18:52:15.062227 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.062209    2579 scope.go:117] "RemoveContainer" containerID="1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e"
Sep 16 18:52:15.063396 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.063379    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"]
Sep 16 18:52:15.068538 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.068514    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6b4888bd86-vt525"]
Sep 16 18:52:15.070987 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.070972    2579 scope.go:117] "RemoveContainer" containerID="5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c"
Sep 16 18:52:15.071266 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:52:15.071249    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c\": container with ID starting with 5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c not found: ID does not exist" containerID="5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c"
Sep 16 18:52:15.071312 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.071273    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c"} err="failed to get container status \"5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c\": rpc error: code = NotFound desc = could not find container \"5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c\": container with ID starting with 5b166f7fd1894851f278251f7b7c971158b7ef33d4dbdb8ee9e6fc306a19329c not found: ID does not exist"
Sep 16 18:52:15.071312 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.071298    2579 scope.go:117] "RemoveContainer" containerID="d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6"
Sep 16 18:52:15.071536 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:52:15.071521    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6\": container with ID starting with d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6 not found: ID does not exist" containerID="d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6"
Sep 16 18:52:15.071579 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.071539    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6"} err="failed to get container status \"d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6\": rpc error: code = NotFound desc = could not find container \"d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6\": container with ID starting with d57270a93e525c3ef91afbfc6285a0328f29507f119f329f12977156a0c376a6 not found: ID does not exist"
Sep 16 18:52:15.071579 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.071551    2579 scope.go:117] "RemoveContainer" containerID="1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e"
Sep 16 18:52:15.071780 ip-10-0-140-67 kubenswrapper[2579]: E0916 18:52:15.071763    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e\": container with ID starting with 1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e not found: ID does not exist" containerID="1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e"
Sep 16 18:52:15.071841 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.071788    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e"} err="failed to get container status \"1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e\": rpc error: code = NotFound desc = could not find container \"1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e\": container with ID starting with 1ef6d53d4aebb82d276f5729999cf579f8bb9b1d835fa4e2057b73f5ed1edb0e not found: ID does not exist"
Sep 16 18:52:15.071841 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.071823    2579 scope.go:117] "RemoveContainer" containerID="d14f290b8916a77b314886f8ddc9b7e960e52d6cf3788c19f579ba8b1a67c378"
Sep 16 18:52:15.078798 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.078780    2579 scope.go:117] "RemoveContainer" containerID="9706edb4e5f61530c6e9c3aeb36739cde64a223536e96fd8c9c5cfc289f991e0"
Sep 16 18:52:15.081606 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.081589    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"]
Sep 16 18:52:15.086398 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.086379    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jvn9k"]
Sep 16 18:52:15.086797 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.086785    2579 scope.go:117] "RemoveContainer" containerID="ab81706ffd2b6d33efd4e2430bcd13533e002225aa40694da3e8cf84dd72a6e4"
Sep 16 18:52:15.811426 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.811395    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" path="/var/lib/kubelet/pods/3ec44839-b801-4b2b-abce-fa4298b8a9c7/volumes"
Sep 16 18:52:15.811886 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.811874    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" path="/var/lib/kubelet/pods/a8edc3c5-2042-4e56-a44c-e3820b3a0264/volumes"
Sep 16 18:52:15.812614 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:15.812600    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" path="/var/lib/kubelet/pods/e56d9a08-e009-4b38-a53c-36fdec39d00a/volumes"
Sep 16 18:52:16.994608 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:16.994579    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:52:16.995019 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:16.994927    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:52:19.805399 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:19.805368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:19.967932 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:19.967885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:26.995878 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:26.995835    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:52:36.995937 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:36.995848    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:52:46.995259 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:46.995213    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:52:56.996084 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:52:56.996044    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 18:53:09.588841 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:53:09.588804    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:09.837036 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:53:09.837006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:41.460215 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:53:41.460177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:35.198858 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:54:35.198826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:35.260700 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:54:35.260674    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:44.058950 ip-10-0-140-67 kubenswrapper[2579]: I0916 18:57:44.058916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:34.644020 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.643980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:34.783519 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.783487    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"]
Sep 16 19:00:34.783768 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.783743    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container" containerID="cri-o://b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13" gracePeriod=30
Sep 16 19:00:34.783850 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.783820    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kube-rbac-proxy" containerID="cri-o://74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056" gracePeriod=30
Sep 16 19:00:34.814169 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.814128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:34.914600 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.914575    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:34.985441 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985408    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"]
Sep 16 19:00:34.985787 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985775    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="storage-initializer"
Sep 16 19:00:34.985844 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985789    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="storage-initializer"
Sep 16 19:00:34.985844 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985801    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container"
Sep 16 19:00:34.985844 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985806    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container"
Sep 16 19:00:34.985844 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985814    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kube-rbac-proxy"
Sep 16 19:00:34.985844 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985820    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kube-rbac-proxy"
Sep 16 19:00:34.985844 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985831    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kube-rbac-proxy"
Sep 16 19:00:34.985844 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985836    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kube-rbac-proxy"
Sep 16 19:00:34.985844 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985843    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kube-rbac-proxy"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985848    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kube-rbac-proxy"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985858    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985865    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985874    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="storage-initializer"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985880    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="storage-initializer"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985885    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985890    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985899    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="storage-initializer"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985904    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="storage-initializer"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985950    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kube-rbac-proxy"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985956    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kube-rbac-proxy"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985965    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kube-rbac-proxy"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985970    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8edc3c5-2042-4e56-a44c-e3820b3a0264" containerName="kserve-container"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985977    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec44839-b801-4b2b-abce-fa4298b8a9c7" containerName="kserve-container"
Sep 16 19:00:34.986070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.985982    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56d9a08-e009-4b38-a53c-36fdec39d00a" containerName="kserve-container"
Sep 16 19:00:34.989027 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.989012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:34.994360 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.994337    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1593f-kube-rbac-proxy-sar-config\""
Sep 16 19:00:34.994474 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:34.994339    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1593f-predictor-serving-cert\""
Sep 16 19:00:35.004645 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.004621    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"]
Sep 16 19:00:35.092526 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.092494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:35.096741 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.096718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-proxy-tls\") pod \"success-200-isvc-1593f-predictor-79647f4cf4-5mc46\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") " pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:35.096861 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.096772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1593f-predictor-79647f4cf4-5mc46\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") " pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:35.096861 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.096845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdkg\" (UniqueName: \"kubernetes.io/projected/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-api-access-cfdkg\") pod \"success-200-isvc-1593f-predictor-79647f4cf4-5mc46\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") " pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:35.121695 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.121669    2579 generic.go:356] "Generic (PLEG): container finished" podID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerID="74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056" exitCode=2
Sep 16 19:00:35.121807 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.121736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" event={"ID":"df06d1f7-4675-4c60-a877-e3c3ac93c475","Type":"ContainerDied","Data":"74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056"}
Sep 16 19:00:35.197867 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.197802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdkg\" (UniqueName: \"kubernetes.io/projected/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-api-access-cfdkg\") pod \"success-200-isvc-1593f-predictor-79647f4cf4-5mc46\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") " pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:35.197983 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.197887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-proxy-tls\") pod \"success-200-isvc-1593f-predictor-79647f4cf4-5mc46\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") " pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:35.197983 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.197917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1593f-predictor-79647f4cf4-5mc46\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") " pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:35.198583 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.198559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1593f-predictor-79647f4cf4-5mc46\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") " pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:35.200401 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.200374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-proxy-tls\") pod \"success-200-isvc-1593f-predictor-79647f4cf4-5mc46\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") " pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:35.201111 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.201090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:35.216597 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.216575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdkg\" (UniqueName: \"kubernetes.io/projected/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-api-access-cfdkg\") pod \"success-200-isvc-1593f-predictor-79647f4cf4-5mc46\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") " pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:35.290614 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.290587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:35.298920 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.298895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:35.583560 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.583524    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"]
Sep 16 19:00:35.587957 ip-10-0-140-67 kubenswrapper[2579]: W0916 19:00:35.587932    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568d7ebd_5b2f_4b3b_8620_21e1a89d1fb5.slice/crio-c570facc2fe25a0e237f7d7ba980c78266618e2edca47f510a235c8d5d184f4d WatchSource:0}: Error finding container c570facc2fe25a0e237f7d7ba980c78266618e2edca47f510a235c8d5d184f4d: Status 404 returned error can't find the container with id c570facc2fe25a0e237f7d7ba980c78266618e2edca47f510a235c8d5d184f4d
Sep 16 19:00:35.643942 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:35.643919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:36.127330 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:36.127297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" event={"ID":"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5","Type":"ContainerStarted","Data":"2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e"}
Sep 16 19:00:36.127754 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:36.127353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" event={"ID":"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5","Type":"ContainerStarted","Data":"ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac"}
Sep 16 19:00:36.127754 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:36.127375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" event={"ID":"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5","Type":"ContainerStarted","Data":"c570facc2fe25a0e237f7d7ba980c78266618e2edca47f510a235c8d5d184f4d"}
Sep 16 19:00:36.127754 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:36.127398    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:36.127754 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:36.127438    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:36.128807 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:36.128783    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:36.164387 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:36.164339    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" podStartSLOduration=2.164325993 podStartE2EDuration="2.164325993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:36.163804175 +0000 UTC m=+1190.829076621" watchObservedRunningTime="2026-09-16 19:00:36.164325993 +0000 UTC m=+1190.829598422"
Sep 16 19:00:37.131856 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:37.131820    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:37.743735 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:37.743696    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 19:00:37.749061 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:37.749037    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 19:00:37.830845 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:37.830822    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 19:00:37.920763 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:37.920678    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df06d1f7-4675-4c60-a877-e3c3ac93c475-proxy-tls\") pod \"df06d1f7-4675-4c60-a877-e3c3ac93c475\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") "
Sep 16 19:00:37.920763 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:37.920725    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwb2g\" (UniqueName: \"kubernetes.io/projected/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-api-access-vwb2g\") pod \"df06d1f7-4675-4c60-a877-e3c3ac93c475\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") "
Sep 16 19:00:37.920763 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:37.920757    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-rbac-proxy-sar-config\") pod \"df06d1f7-4675-4c60-a877-e3c3ac93c475\" (UID: \"df06d1f7-4675-4c60-a877-e3c3ac93c475\") "
Sep 16 19:00:37.921246 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:37.921213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df06d1f7-4675-4c60-a877-e3c3ac93c475" (UID: "df06d1f7-4675-4c60-a877-e3c3ac93c475"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:37.923063 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:37.923042    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df06d1f7-4675-4c60-a877-e3c3ac93c475-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df06d1f7-4675-4c60-a877-e3c3ac93c475" (UID: "df06d1f7-4675-4c60-a877-e3c3ac93c475"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:37.923219 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:37.923196    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-api-access-vwb2g" (OuterVolumeSpecName: "kube-api-access-vwb2g") pod "df06d1f7-4675-4c60-a877-e3c3ac93c475" (UID: "df06d1f7-4675-4c60-a877-e3c3ac93c475"). InnerVolumeSpecName "kube-api-access-vwb2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:38.022273 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.022232    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df06d1f7-4675-4c60-a877-e3c3ac93c475-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:00:38.022273 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.022269    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwb2g\" (UniqueName: \"kubernetes.io/projected/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-api-access-vwb2g\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:00:38.022273 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.022279    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df06d1f7-4675-4c60-a877-e3c3ac93c475-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:00:38.140133 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.140098    2579 generic.go:356] "Generic (PLEG): container finished" podID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerID="b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13" exitCode=0
Sep 16 19:00:38.140619 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.140187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"
Sep 16 19:00:38.140619 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.140186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" event={"ID":"df06d1f7-4675-4c60-a877-e3c3ac93c475","Type":"ContainerDied","Data":"b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13"}
Sep 16 19:00:38.140619 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.140316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn" event={"ID":"df06d1f7-4675-4c60-a877-e3c3ac93c475","Type":"ContainerDied","Data":"e0f76e8767d7e13385147a529af6e880e9e50af547abdf2a9544d3ad7440f2bb"}
Sep 16 19:00:38.140619 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.140343    2579 scope.go:117] "RemoveContainer" containerID="74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056"
Sep 16 19:00:38.148971 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.148948    2579 scope.go:117] "RemoveContainer" containerID="b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13"
Sep 16 19:00:38.156575 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.156558    2579 scope.go:117] "RemoveContainer" containerID="74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056"
Sep 16 19:00:38.156834 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:00:38.156816    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056\": container with ID starting with 74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056 not found: ID does not exist" containerID="74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056"
Sep 16 19:00:38.156884 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.156843    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056"} err="failed to get container status \"74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056\": rpc error: code = NotFound desc = could not find container \"74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056\": container with ID starting with 74956a0269fbb3893057ff631af25c2bd5bb6d9b2e7a091b081dbfc9f98a9056 not found: ID does not exist"
Sep 16 19:00:38.156884 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.156865    2579 scope.go:117] "RemoveContainer" containerID="b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13"
Sep 16 19:00:38.157097 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:00:38.157082    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13\": container with ID starting with b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13 not found: ID does not exist" containerID="b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13"
Sep 16 19:00:38.157133 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.157103    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13"} err="failed to get container status \"b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13\": rpc error: code = NotFound desc = could not find container \"b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13\": container with ID starting with b1cd1605e2816fe78d8bb097e10402b660383d18c8b0839a4a3f18a605a95e13 not found: ID does not exist"
Sep 16 19:00:38.170819 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.170757    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"]
Sep 16 19:00:38.182202 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:38.182180    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15f71-predictor-784478884b-fqkdn"]
Sep 16 19:00:39.811839 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:39.811804    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" path="/var/lib/kubelet/pods/df06d1f7-4675-4c60-a877-e3c3ac93c475/volumes"
Sep 16 19:00:42.136238 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:42.136211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:00:42.136755 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:42.136723    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:52.137146 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:00:52.137102    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:01:02.136935 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:02.136891    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:01:12.137212 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:12.137173    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:01:22.138142 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:22.138111    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:01:24.201328 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.201287    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:24.226717 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.226687    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"]
Sep 16 19:01:24.227040 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.227003    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container" containerID="cri-o://09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e" gracePeriod=30
Sep 16 19:01:24.227202 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.227034    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kube-rbac-proxy" containerID="cri-o://b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62" gracePeriod=30
Sep 16 19:01:24.273252 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.273219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:24.380278 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.380244    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"]
Sep 16 19:01:24.380630 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.380613    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container"
Sep 16 19:01:24.380630 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.380626    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container"
Sep 16 19:01:24.380708 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.380637    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kube-rbac-proxy"
Sep 16 19:01:24.380708 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.380642    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kube-rbac-proxy"
Sep 16 19:01:24.380708 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.380706    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kube-rbac-proxy"
Sep 16 19:01:24.380800 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.380715    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df06d1f7-4675-4c60-a877-e3c3ac93c475" containerName="kserve-container"
Sep 16 19:01:24.383831 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.383815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.386610 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.386585    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-34033-kube-rbac-proxy-sar-config\""
Sep 16 19:01:24.386724 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.386625    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-34033-predictor-serving-cert\""
Sep 16 19:01:24.398705 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.398682    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"]
Sep 16 19:01:24.417799 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.417765    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:24.509550 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.509474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:24.522900 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.522860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-34033-predictor-6dcb7cd944-jm6v8\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") " pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.523046 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.522988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8325601-b77d-460c-9a44-13b09ff6e0b7-proxy-tls\") pod \"success-200-isvc-34033-predictor-6dcb7cd944-jm6v8\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") " pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.523109 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.523058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7f7r\" (UniqueName: \"kubernetes.io/projected/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-api-access-f7f7r\") pod \"success-200-isvc-34033-predictor-6dcb7cd944-jm6v8\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") " pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.589343 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.589305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:24.624370 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.624327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8325601-b77d-460c-9a44-13b09ff6e0b7-proxy-tls\") pod \"success-200-isvc-34033-predictor-6dcb7cd944-jm6v8\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") " pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.624605 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.624384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7f7r\" (UniqueName: \"kubernetes.io/projected/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-api-access-f7f7r\") pod \"success-200-isvc-34033-predictor-6dcb7cd944-jm6v8\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") " pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.624605 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.624434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-34033-predictor-6dcb7cd944-jm6v8\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") " pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.625050 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.625032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-34033-predictor-6dcb7cd944-jm6v8\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") " pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.626951 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.626930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8325601-b77d-460c-9a44-13b09ff6e0b7-proxy-tls\") pod \"success-200-isvc-34033-predictor-6dcb7cd944-jm6v8\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") " pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.637032 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.637005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7f7r\" (UniqueName: \"kubernetes.io/projected/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-api-access-f7f7r\") pod \"success-200-isvc-34033-predictor-6dcb7cd944-jm6v8\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") " pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.693780 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.693750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:24.860964 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.860931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:24.985480 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:24.985292    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"]
Sep 16 19:01:24.988282 ip-10-0-140-67 kubenswrapper[2579]: W0916 19:01:24.988254    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8325601_b77d_460c_9a44_13b09ff6e0b7.slice/crio-28fbf065bc3dc0db5237d82c85dd3ab062f5a412cd22abcf089ae4488516b3a8 WatchSource:0}: Error finding container 28fbf065bc3dc0db5237d82c85dd3ab062f5a412cd22abcf089ae4488516b3a8: Status 404 returned error can't find the container with id 28fbf065bc3dc0db5237d82c85dd3ab062f5a412cd22abcf089ae4488516b3a8
Sep 16 19:01:25.343384 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:25.343353    2579 generic.go:356] "Generic (PLEG): container finished" podID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerID="b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62" exitCode=2
Sep 16 19:01:25.343800 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:25.343427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" event={"ID":"242b5f6f-7683-4f38-8ae8-1e17a985de8d","Type":"ContainerDied","Data":"b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62"}
Sep 16 19:01:25.345197 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:25.345175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" event={"ID":"c8325601-b77d-460c-9a44-13b09ff6e0b7","Type":"ContainerStarted","Data":"c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1"}
Sep 16 19:01:25.345321 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:25.345224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" event={"ID":"c8325601-b77d-460c-9a44-13b09ff6e0b7","Type":"ContainerStarted","Data":"a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72"}
Sep 16 19:01:25.345321 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:25.345250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" event={"ID":"c8325601-b77d-460c-9a44-13b09ff6e0b7","Type":"ContainerStarted","Data":"28fbf065bc3dc0db5237d82c85dd3ab062f5a412cd22abcf089ae4488516b3a8"}
Sep 16 19:01:25.345321 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:25.345308    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:25.373498 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:25.373449    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" podStartSLOduration=1.3734342769999999 podStartE2EDuration="1.373434277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:25.370775219 +0000 UTC m=+1240.036047654" watchObservedRunningTime="2026-09-16 19:01:25.373434277 +0000 UTC m=+1240.038706706"
Sep 16 19:01:26.352722 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:26.352691    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:26.354044 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:26.354014    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:26.991363 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:26.991318    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 16 19:01:26.995686 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:26.995653    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 19:01:27.357694 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.357661    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:27.570722 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.570691    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 19:01:27.654539 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.654451    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/242b5f6f-7683-4f38-8ae8-1e17a985de8d-proxy-tls\") pod \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") "
Sep 16 19:01:27.654698 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.654568    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-rbac-proxy-sar-config\") pod \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") "
Sep 16 19:01:27.654698 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.654615    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn9m5\" (UniqueName: \"kubernetes.io/projected/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-api-access-zn9m5\") pod \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\" (UID: \"242b5f6f-7683-4f38-8ae8-1e17a985de8d\") "
Sep 16 19:01:27.654971 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.654942    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "242b5f6f-7683-4f38-8ae8-1e17a985de8d" (UID: "242b5f6f-7683-4f38-8ae8-1e17a985de8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:27.656695 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.656667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242b5f6f-7683-4f38-8ae8-1e17a985de8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "242b5f6f-7683-4f38-8ae8-1e17a985de8d" (UID: "242b5f6f-7683-4f38-8ae8-1e17a985de8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:27.656802 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.656785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-api-access-zn9m5" (OuterVolumeSpecName: "kube-api-access-zn9m5") pod "242b5f6f-7683-4f38-8ae8-1e17a985de8d" (UID: "242b5f6f-7683-4f38-8ae8-1e17a985de8d"). InnerVolumeSpecName "kube-api-access-zn9m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:27.756061 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.756023    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:01:27.756061 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.756064    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9m5\" (UniqueName: \"kubernetes.io/projected/242b5f6f-7683-4f38-8ae8-1e17a985de8d-kube-api-access-zn9m5\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:01:27.756258 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:27.756081    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/242b5f6f-7683-4f38-8ae8-1e17a985de8d-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:01:28.363770 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.363735    2579 generic.go:356] "Generic (PLEG): container finished" podID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerID="09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e" exitCode=0
Sep 16 19:01:28.364125 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.363782    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"
Sep 16 19:01:28.364125 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.363809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" event={"ID":"242b5f6f-7683-4f38-8ae8-1e17a985de8d","Type":"ContainerDied","Data":"09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e"}
Sep 16 19:01:28.364125 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.363858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr" event={"ID":"242b5f6f-7683-4f38-8ae8-1e17a985de8d","Type":"ContainerDied","Data":"cc7b50e846cf8585a0cd626dc905230abe0280ce992aaa166b22516de3c9c908"}
Sep 16 19:01:28.364125 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.363887    2579 scope.go:117] "RemoveContainer" containerID="b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62"
Sep 16 19:01:28.372450 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.372435    2579 scope.go:117] "RemoveContainer" containerID="09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e"
Sep 16 19:01:28.380036 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.380016    2579 scope.go:117] "RemoveContainer" containerID="b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62"
Sep 16 19:01:28.380355 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:01:28.380335    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62\": container with ID starting with b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62 not found: ID does not exist" containerID="b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62"
Sep 16 19:01:28.380431 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.380364    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62"} err="failed to get container status \"b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62\": rpc error: code = NotFound desc = could not find container \"b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62\": container with ID starting with b998da4325abdc2e9a1deb172ef58c616cc4a1775d9ac812406705798b617e62 not found: ID does not exist"
Sep 16 19:01:28.380431 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.380381    2579 scope.go:117] "RemoveContainer" containerID="09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e"
Sep 16 19:01:28.380681 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:01:28.380662    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e\": container with ID starting with 09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e not found: ID does not exist" containerID="09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e"
Sep 16 19:01:28.380733 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.380687    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e"} err="failed to get container status \"09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e\": rpc error: code = NotFound desc = could not find container \"09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e\": container with ID starting with 09781e80bdc5dbdce3fbb1dedba316f18d73abb79bf3544781ed59640bfee02e not found: ID does not exist"
Sep 16 19:01:28.391480 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.391456    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"]
Sep 16 19:01:28.402614 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:28.402590    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21301-predictor-68796d9f44-xxrjr"]
Sep 16 19:01:29.811268 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:29.811235    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" path="/var/lib/kubelet/pods/242b5f6f-7683-4f38-8ae8-1e17a985de8d/volumes"
Sep 16 19:01:32.362975 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:32.362947    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:01:32.363524 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:32.363497    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:34.821309 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:34.821272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:34.912348 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:34.912312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:42.363817 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:42.363727    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:44.941330 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:44.941298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:45.101359 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.101330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:45.116009 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.115976    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"]
Sep 16 19:01:45.116233 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.116208    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kserve-container" containerID="cri-o://ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac" gracePeriod=30
Sep 16 19:01:45.116319 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.116255    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kube-rbac-proxy" containerID="cri-o://2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e" gracePeriod=30
Sep 16 19:01:45.202011 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.201928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:45.278668 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.278631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:45.280604 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.280580    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"]
Sep 16 19:01:45.280939 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.280927    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container"
Sep 16 19:01:45.280985 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.280941    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container"
Sep 16 19:01:45.280985 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.280952    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kube-rbac-proxy"
Sep 16 19:01:45.280985 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.280957    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kube-rbac-proxy"
Sep 16 19:01:45.281076 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.281026    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kube-rbac-proxy"
Sep 16 19:01:45.281076 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.281035    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="242b5f6f-7683-4f38-8ae8-1e17a985de8d" containerName="kserve-container"
Sep 16 19:01:45.284085 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.284070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:45.286837 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.286816    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e4920-predictor-serving-cert\""
Sep 16 19:01:45.287324 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.287305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e4920-kube-rbac-proxy-sar-config\""
Sep 16 19:01:45.298458 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.298437    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"]
Sep 16 19:01:45.311056 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.311033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-proxy-tls\") pod \"success-200-isvc-e4920-predictor-868cb5f55d-t4824\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") " pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:45.311143 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.311071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e4920-predictor-868cb5f55d-t4824\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") " pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:45.311205 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.311175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shsjp\" (UniqueName: \"kubernetes.io/projected/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-api-access-shsjp\") pod \"success-200-isvc-e4920-predictor-868cb5f55d-t4824\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") " pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:45.411862 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.411827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-proxy-tls\") pod \"success-200-isvc-e4920-predictor-868cb5f55d-t4824\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") " pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:45.411862 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.411867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e4920-predictor-868cb5f55d-t4824\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") " pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:45.412116 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.411924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shsjp\" (UniqueName: \"kubernetes.io/projected/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-api-access-shsjp\") pod \"success-200-isvc-e4920-predictor-868cb5f55d-t4824\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") " pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:45.412116 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:01:45.412006    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e4920-predictor-serving-cert: secret "success-200-isvc-e4920-predictor-serving-cert" not found
Sep 16 19:01:45.412116 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:01:45.412094    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-proxy-tls podName:63e0cf4f-8afb-4e8c-81bf-0aad239c6876 nodeName:}" failed. No retries permitted until 2026-09-16 19:01:45.912071334 +0000 UTC m=+1260.577343757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-proxy-tls") pod "success-200-isvc-e4920-predictor-868cb5f55d-t4824" (UID: "63e0cf4f-8afb-4e8c-81bf-0aad239c6876") : secret "success-200-isvc-e4920-predictor-serving-cert" not found
Sep 16 19:01:45.412599 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.412579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e4920-predictor-868cb5f55d-t4824\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") " pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:45.423800 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.423772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:45.431649 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.431622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shsjp\" (UniqueName: \"kubernetes.io/projected/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-api-access-shsjp\") pod \"success-200-isvc-e4920-predictor-868cb5f55d-t4824\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") " pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:45.440220 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.440191    2579 generic.go:356] "Generic (PLEG): container finished" podID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerID="2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e" exitCode=2
Sep 16 19:01:45.440358 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.440259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" event={"ID":"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5","Type":"ContainerDied","Data":"2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e"}
Sep 16 19:01:45.516506 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.516429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:45.698024 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.697989    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:45.915499 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.915461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-proxy-tls\") pod \"success-200-isvc-e4920-predictor-868cb5f55d-t4824\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") " pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:45.917856 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:45.917832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-proxy-tls\") pod \"success-200-isvc-e4920-predictor-868cb5f55d-t4824\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") " pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:46.193893 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:46.193816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:46.482110 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:46.482029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"]
Sep 16 19:01:46.485202 ip-10-0-140-67 kubenswrapper[2579]: W0916 19:01:46.485173    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e0cf4f_8afb_4e8c_81bf_0aad239c6876.slice/crio-dd072566bc647f090d024af6d45f787792c481dbcc4d434b42b435e1dd1bfa01 WatchSource:0}: Error finding container dd072566bc647f090d024af6d45f787792c481dbcc4d434b42b435e1dd1bfa01: Status 404 returned error can't find the container with id dd072566bc647f090d024af6d45f787792c481dbcc4d434b42b435e1dd1bfa01
Sep 16 19:01:47.132104 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:47.132062    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 16 19:01:47.450732 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:47.450652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" event={"ID":"63e0cf4f-8afb-4e8c-81bf-0aad239c6876","Type":"ContainerStarted","Data":"d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8"}
Sep 16 19:01:47.450732 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:47.450705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" event={"ID":"63e0cf4f-8afb-4e8c-81bf-0aad239c6876","Type":"ContainerStarted","Data":"d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981"}
Sep 16 19:01:47.450732 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:47.450725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" event={"ID":"63e0cf4f-8afb-4e8c-81bf-0aad239c6876","Type":"ContainerStarted","Data":"dd072566bc647f090d024af6d45f787792c481dbcc4d434b42b435e1dd1bfa01"}
Sep 16 19:01:47.451174 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:47.450788    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:47.484240 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:47.484183    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" podStartSLOduration=2.484168386 podStartE2EDuration="2.484168386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:47.483358176 +0000 UTC m=+1262.148630622" watchObservedRunningTime="2026-09-16 19:01:47.484168386 +0000 UTC m=+1262.149440807"
Sep 16 19:01:48.372091 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.372067    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:01:48.439144 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.439054    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-proxy-tls\") pod \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") "
Sep 16 19:01:48.439144 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.439146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfdkg\" (UniqueName: \"kubernetes.io/projected/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-api-access-cfdkg\") pod \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") "
Sep 16 19:01:48.439405 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.439200    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-rbac-proxy-sar-config\") pod \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\" (UID: \"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5\") "
Sep 16 19:01:48.439538 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.439514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" (UID: "568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:48.441340 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.441317    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-api-access-cfdkg" (OuterVolumeSpecName: "kube-api-access-cfdkg") pod "568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" (UID: "568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5"). InnerVolumeSpecName "kube-api-access-cfdkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:48.441431 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.441361    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" (UID: "568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:48.457599 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.457570    2579 generic.go:356] "Generic (PLEG): container finished" podID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerID="ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac" exitCode=0
Sep 16 19:01:48.457998 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.457609    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" event={"ID":"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5","Type":"ContainerDied","Data":"ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac"}
Sep 16 19:01:48.457998 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.457688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46" event={"ID":"568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5","Type":"ContainerDied","Data":"c570facc2fe25a0e237f7d7ba980c78266618e2edca47f510a235c8d5d184f4d"}
Sep 16 19:01:48.457998 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.457620    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"
Sep 16 19:01:48.457998 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.457734    2579 scope.go:117] "RemoveContainer" containerID="2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e"
Sep 16 19:01:48.458419 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.458350    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:48.459922 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.459893    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:01:48.467123 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.467106    2579 scope.go:117] "RemoveContainer" containerID="ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac"
Sep 16 19:01:48.474929 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.474911    2579 scope.go:117] "RemoveContainer" containerID="2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e"
Sep 16 19:01:48.475203 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:01:48.475187    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e\": container with ID starting with 2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e not found: ID does not exist" containerID="2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e"
Sep 16 19:01:48.475247 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.475211    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e"} err="failed to get container status \"2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e\": rpc error: code = NotFound desc = could not find container \"2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e\": container with ID starting with 2a9ea775e7ae0cf70077a8788e583c35e7e5b09e4454e335b0733a8f62f99f8e not found: ID does not exist"
Sep 16 19:01:48.475247 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.475226    2579 scope.go:117] "RemoveContainer" containerID="ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac"
Sep 16 19:01:48.475469 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:01:48.475452    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac\": container with ID starting with ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac not found: ID does not exist" containerID="ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac"
Sep 16 19:01:48.475505 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.475475    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac"} err="failed to get container status \"ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac\": rpc error: code = NotFound desc = could not find container \"ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac\": container with ID starting with ddd45d66072c287189ecadee4e45b6c6b2db5684f227e1476b61cc3d3b0aa1ac not found: ID does not exist"
Sep 16 19:01:48.490568 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.490547    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"]
Sep 16 19:01:48.498004 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.497984    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1593f-predictor-79647f4cf4-5mc46"]
Sep 16 19:01:48.540020 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.540000    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.540138 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.540026    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdkg\" (UniqueName: \"kubernetes.io/projected/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-api-access-cfdkg\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.540138 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:48.540041    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:01:49.463984 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:49.463943    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:01:49.811237 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:49.811141    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" path="/var/lib/kubelet/pods/568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5/volumes"
Sep 16 19:01:52.363834 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:52.363786    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:54.469576 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:54.469550    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:01:54.469960 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:01:54.469935    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:02:02.363957 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:02.363913    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:04.469961 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:04.469918    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:02:12.364866 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:12.364836    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:02:14.470471 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:14.470427    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:02:24.372416 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:24.372386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:24.470779 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:24.470741    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:02:24.477057 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:24.477033    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:34.471349 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.471319    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:02:34.522973 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.522937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:34.686012 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.685979    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"]
Sep 16 19:02:34.686292 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.686268    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kserve-container" containerID="cri-o://a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72" gracePeriod=30
Sep 16 19:02:34.686403 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.686282    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kube-rbac-proxy" containerID="cri-o://c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1" gracePeriod=30
Sep 16 19:02:34.699379 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.699349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:34.802666 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.802584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:34.903142 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.903109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:34.910675 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.910648    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"]
Sep 16 19:02:34.911013 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.911001    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kserve-container"
Sep 16 19:02:34.911057 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.911015    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kserve-container"
Sep 16 19:02:34.911057 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.911032    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kube-rbac-proxy"
Sep 16 19:02:34.911057 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.911037    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kube-rbac-proxy"
Sep 16 19:02:34.911164 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.911096    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kserve-container"
Sep 16 19:02:34.911164 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.911106    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="568d7ebd-5b2f-4b3b-8620-21e1a89d1fb5" containerName="kube-rbac-proxy"
Sep 16 19:02:34.914221 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.914203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:34.917981 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.917956    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e3b29-kube-rbac-proxy-sar-config\""
Sep 16 19:02:34.918334 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.918318    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e3b29-predictor-serving-cert\""
Sep 16 19:02:34.932391 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.932373    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"]
Sep 16 19:02:34.942371 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.942352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") " pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:34.942469 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.942377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vfs8\" (UniqueName: \"kubernetes.io/projected/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-api-access-7vfs8\") pod \"success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") " pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:34.942469 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:34.942403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-proxy-tls\") pod \"success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") " pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:35.001093 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.001065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:35.043466 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.043445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") " pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:35.043466 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.043472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vfs8\" (UniqueName: \"kubernetes.io/projected/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-api-access-7vfs8\") pod \"success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") " pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:35.043620 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.043498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-proxy-tls\") pod \"success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") " pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:35.044070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.044051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") " pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:35.046205 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.046186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-proxy-tls\") pod \"success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") " pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:35.058360 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.058306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vfs8\" (UniqueName: \"kubernetes.io/projected/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-api-access-7vfs8\") pod \"success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") " pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:35.130647 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.130612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:35.223532 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.223500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:35.439060 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.439031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:35.511271 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.511249    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"]
Sep 16 19:02:35.513867 ip-10-0-140-67 kubenswrapper[2579]: W0916 19:02:35.513836    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2db5de4_8f0a_428f_b8ab_435f68d74cdc.slice/crio-055e1e3766e16f14a10e990b422ae20999238faff7b974ba21a0b0dfcfbbd6d0 WatchSource:0}: Error finding container 055e1e3766e16f14a10e990b422ae20999238faff7b974ba21a0b0dfcfbbd6d0: Status 404 returned error can't find the container with id 055e1e3766e16f14a10e990b422ae20999238faff7b974ba21a0b0dfcfbbd6d0
Sep 16 19:02:35.662867 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.662840    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerID="c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1" exitCode=2
Sep 16 19:02:35.662990 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.662892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" event={"ID":"c8325601-b77d-460c-9a44-13b09ff6e0b7","Type":"ContainerDied","Data":"c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1"}
Sep 16 19:02:35.664979 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.664952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" event={"ID":"d2db5de4-8f0a-428f-b8ab-435f68d74cdc","Type":"ContainerStarted","Data":"3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447"}
Sep 16 19:02:35.665093 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.664993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" event={"ID":"d2db5de4-8f0a-428f-b8ab-435f68d74cdc","Type":"ContainerStarted","Data":"40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a"}
Sep 16 19:02:35.665093 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.665013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" event={"ID":"d2db5de4-8f0a-428f-b8ab-435f68d74cdc","Type":"ContainerStarted","Data":"055e1e3766e16f14a10e990b422ae20999238faff7b974ba21a0b0dfcfbbd6d0"}
Sep 16 19:02:35.665093 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.665037    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:35.665278 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.665141    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:35.666415 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.666395    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:02:35.692066 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:35.691978    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podStartSLOduration=1.691962561 podStartE2EDuration="1.691962561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:35.691530232 +0000 UTC m=+1310.356802682" watchObservedRunningTime="2026-09-16 19:02:35.691962561 +0000 UTC m=+1310.357234991"
Sep 16 19:02:36.669783 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:36.669750    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:02:37.357827 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.357792    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 19:02:37.635354 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.635331    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:02:37.663932 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.663906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-rbac-proxy-sar-config\") pod \"c8325601-b77d-460c-9a44-13b09ff6e0b7\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") "
Sep 16 19:02:37.664052 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.663964    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8325601-b77d-460c-9a44-13b09ff6e0b7-proxy-tls\") pod \"c8325601-b77d-460c-9a44-13b09ff6e0b7\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") "
Sep 16 19:02:37.664112 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.664052    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7f7r\" (UniqueName: \"kubernetes.io/projected/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-api-access-f7f7r\") pod \"c8325601-b77d-460c-9a44-13b09ff6e0b7\" (UID: \"c8325601-b77d-460c-9a44-13b09ff6e0b7\") "
Sep 16 19:02:37.664356 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.664322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8325601-b77d-460c-9a44-13b09ff6e0b7" (UID: "c8325601-b77d-460c-9a44-13b09ff6e0b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:37.666493 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.666466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8325601-b77d-460c-9a44-13b09ff6e0b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8325601-b77d-460c-9a44-13b09ff6e0b7" (UID: "c8325601-b77d-460c-9a44-13b09ff6e0b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:37.667286 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.667260    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-api-access-f7f7r" (OuterVolumeSpecName: "kube-api-access-f7f7r") pod "c8325601-b77d-460c-9a44-13b09ff6e0b7" (UID: "c8325601-b77d-460c-9a44-13b09ff6e0b7"). InnerVolumeSpecName "kube-api-access-f7f7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:37.676470 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.676438    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerID="a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72" exitCode=0
Sep 16 19:02:37.676823 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.676487    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"
Sep 16 19:02:37.676823 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.676522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" event={"ID":"c8325601-b77d-460c-9a44-13b09ff6e0b7","Type":"ContainerDied","Data":"a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72"}
Sep 16 19:02:37.676823 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.676587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8" event={"ID":"c8325601-b77d-460c-9a44-13b09ff6e0b7","Type":"ContainerDied","Data":"28fbf065bc3dc0db5237d82c85dd3ab062f5a412cd22abcf089ae4488516b3a8"}
Sep 16 19:02:37.676823 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.676628    2579 scope.go:117] "RemoveContainer" containerID="c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1"
Sep 16 19:02:37.685960 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.685943    2579 scope.go:117] "RemoveContainer" containerID="a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72"
Sep 16 19:02:37.693589 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.693573    2579 scope.go:117] "RemoveContainer" containerID="c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1"
Sep 16 19:02:37.693845 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:02:37.693827    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1\": container with ID starting with c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1 not found: ID does not exist" containerID="c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1"
Sep 16 19:02:37.693886 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.693853    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1"} err="failed to get container status \"c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1\": rpc error: code = NotFound desc = could not find container \"c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1\": container with ID starting with c779cea3158b610c6671fae5af11e4234384ec5bcab331d095dc845a6879ddd1 not found: ID does not exist"
Sep 16 19:02:37.693886 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.693870    2579 scope.go:117] "RemoveContainer" containerID="a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72"
Sep 16 19:02:37.694112 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:02:37.694099    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72\": container with ID starting with a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72 not found: ID does not exist" containerID="a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72"
Sep 16 19:02:37.694180 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.694115    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72"} err="failed to get container status \"a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72\": rpc error: code = NotFound desc = could not find container \"a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72\": container with ID starting with a97f5cd95b37f083e70be3ec3d4604852ecf5e129bfbd5ec5413374351db7c72 not found: ID does not exist"
Sep 16 19:02:37.707374 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.707354    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"]
Sep 16 19:02:37.718283 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.718262    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34033-predictor-6dcb7cd944-jm6v8"]
Sep 16 19:02:37.765076 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.765058    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8325601-b77d-460c-9a44-13b09ff6e0b7-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:02:37.765076 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.765078    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7f7r\" (UniqueName: \"kubernetes.io/projected/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-api-access-f7f7r\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:02:37.765209 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.765087    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8325601-b77d-460c-9a44-13b09ff6e0b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:02:37.811461 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:37.811441    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" path="/var/lib/kubelet/pods/c8325601-b77d-460c-9a44-13b09ff6e0b7/volumes"
Sep 16 19:02:41.674282 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:41.674247    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:02:41.674884 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:41.674857    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:02:45.197281 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:45.197249    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:45.545451 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:45.545376    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:50.406124 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:50.406083    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:51.675729 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:02:51.675691    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:01.675041 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:03:01.675000    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:11.674911 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:03:11.674873    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:21.675359 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:03:21.675328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:03:34.785964 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:03:34.785927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:34.996684 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:03:34.996656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:34.935850 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:04:34.935804    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:45.401347 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:08:45.401314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:49.310182 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:08:49.310133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:00.455437 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:09:00.455408    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:10.234597 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:09:10.234512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:00.005340 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.005298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:00.153690 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.153659    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"]
Sep 16 19:11:00.153917 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.153885    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kserve-container" containerID="cri-o://d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981" gracePeriod=30
Sep 16 19:11:00.154070 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.153936    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kube-rbac-proxy" containerID="cri-o://d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8" gracePeriod=30
Sep 16 19:11:00.181303 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.181279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:00.309248 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.309191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:00.378522 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.378487    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"]
Sep 16 19:11:00.378861 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.378848    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kserve-container"
Sep 16 19:11:00.378912 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.378863    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kserve-container"
Sep 16 19:11:00.378912 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.378883    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kube-rbac-proxy"
Sep 16 19:11:00.378912 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.378889    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kube-rbac-proxy"
Sep 16 19:11:00.379020 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.378938    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kserve-container"
Sep 16 19:11:00.379020 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.378947    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8325601-b77d-460c-9a44-13b09ff6e0b7" containerName="kube-rbac-proxy"
Sep 16 19:11:00.381982 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.381964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.387203 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.387182    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-617e5-predictor-serving-cert\""
Sep 16 19:11:00.387307 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.387216    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-617e5-kube-rbac-proxy-sar-config\""
Sep 16 19:11:00.409889 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.409857    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"]
Sep 16 19:11:00.488969 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.488938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-proxy-tls\") pod \"success-200-isvc-617e5-predictor-776d98c8-5r9t7\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") " pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.488969 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.488973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpj24\" (UniqueName: \"kubernetes.io/projected/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-api-access-cpj24\") pod \"success-200-isvc-617e5-predictor-776d98c8-5r9t7\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") " pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.489189 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.488995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-617e5-predictor-776d98c8-5r9t7\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") " pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.551859 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.551787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:00.589516 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.589483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-proxy-tls\") pod \"success-200-isvc-617e5-predictor-776d98c8-5r9t7\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") " pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.589654 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.589523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpj24\" (UniqueName: \"kubernetes.io/projected/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-api-access-cpj24\") pod \"success-200-isvc-617e5-predictor-776d98c8-5r9t7\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") " pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.589796 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.589776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-617e5-predictor-776d98c8-5r9t7\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") " pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.590555 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.590533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-617e5-predictor-776d98c8-5r9t7\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") " pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.592291 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.592275    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-proxy-tls\") pod \"success-200-isvc-617e5-predictor-776d98c8-5r9t7\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") " pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.625039 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.625021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpj24\" (UniqueName: \"kubernetes.io/projected/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-api-access-cpj24\") pod \"success-200-isvc-617e5-predictor-776d98c8-5r9t7\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") " pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.667018 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.666985    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:00.692201 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.692177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:00.806535 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.806454    2579 generic.go:356] "Generic (PLEG): container finished" podID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerID="d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8" exitCode=2
Sep 16 19:11:00.806668 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.806532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" event={"ID":"63e0cf4f-8afb-4e8c-81bf-0aad239c6876","Type":"ContainerDied","Data":"d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8"}
Sep 16 19:11:00.980504 ip-10-0-140-67 kubenswrapper[2579]: W0916 19:11:00.980474    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf903b59d_cf5a_496c_9bd3_1ca24dd9f7d3.slice/crio-cf23ba6d9a0e36d215cb4c3742b42bcab7a94b3239a0d8f8a0dc67ffb283ea5e WatchSource:0}: Error finding container cf23ba6d9a0e36d215cb4c3742b42bcab7a94b3239a0d8f8a0dc67ffb283ea5e: Status 404 returned error can't find the container with id cf23ba6d9a0e36d215cb4c3742b42bcab7a94b3239a0d8f8a0dc67ffb283ea5e
Sep 16 19:11:00.982016 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:00.981963    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"]
Sep 16 19:11:01.211615 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:01.211578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:01.812226 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:01.812182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" event={"ID":"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3","Type":"ContainerStarted","Data":"497d98c5337afa5264a9220432a1c2f1e641567f3322e3c4ef10d2b8d28c8732"}
Sep 16 19:11:01.812392 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:01.812245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" event={"ID":"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3","Type":"ContainerStarted","Data":"1cf494eb8b40d3c33dc53b00072ee4c72e965995ac118111e3f435b0e56a32b4"}
Sep 16 19:11:01.812392 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:01.812351    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:01.812498 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:01.812484    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:01.812539 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:01.812521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" event={"ID":"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3","Type":"ContainerStarted","Data":"cf23ba6d9a0e36d215cb4c3742b42bcab7a94b3239a0d8f8a0dc67ffb283ea5e"}
Sep 16 19:11:01.813508 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:01.813478    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:11:01.857601 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:01.857559    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podStartSLOduration=1.857543419 podStartE2EDuration="1.857543419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:01.856480654 +0000 UTC m=+1816.521753105" watchObservedRunningTime="2026-09-16 19:11:01.857543419 +0000 UTC m=+1816.522815850"
Sep 16 19:11:02.817281 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:02.817246    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:11:03.101408 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.101388    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:11:03.212446 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.212419    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-rbac-proxy-sar-config\") pod \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") "
Sep 16 19:11:03.212604 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.212465    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-proxy-tls\") pod \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") "
Sep 16 19:11:03.212604 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.212498    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shsjp\" (UniqueName: \"kubernetes.io/projected/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-api-access-shsjp\") pod \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\" (UID: \"63e0cf4f-8afb-4e8c-81bf-0aad239c6876\") "
Sep 16 19:11:03.212774 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.212749    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "63e0cf4f-8afb-4e8c-81bf-0aad239c6876" (UID: "63e0cf4f-8afb-4e8c-81bf-0aad239c6876"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:03.214769 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.214744    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63e0cf4f-8afb-4e8c-81bf-0aad239c6876" (UID: "63e0cf4f-8afb-4e8c-81bf-0aad239c6876"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:03.214769 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.214756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-api-access-shsjp" (OuterVolumeSpecName: "kube-api-access-shsjp") pod "63e0cf4f-8afb-4e8c-81bf-0aad239c6876" (UID: "63e0cf4f-8afb-4e8c-81bf-0aad239c6876"). InnerVolumeSpecName "kube-api-access-shsjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:03.313480 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.313457    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:11:03.313480 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.313477    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:11:03.313614 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.313488    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shsjp\" (UniqueName: \"kubernetes.io/projected/63e0cf4f-8afb-4e8c-81bf-0aad239c6876-kube-api-access-shsjp\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:11:03.822520 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.822491    2579 generic.go:356] "Generic (PLEG): container finished" podID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerID="d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981" exitCode=0
Sep 16 19:11:03.822935 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.822550    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"
Sep 16 19:11:03.822935 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.822573    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" event={"ID":"63e0cf4f-8afb-4e8c-81bf-0aad239c6876","Type":"ContainerDied","Data":"d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981"}
Sep 16 19:11:03.822935 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.822627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824" event={"ID":"63e0cf4f-8afb-4e8c-81bf-0aad239c6876","Type":"ContainerDied","Data":"dd072566bc647f090d024af6d45f787792c481dbcc4d434b42b435e1dd1bfa01"}
Sep 16 19:11:03.822935 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.822657    2579 scope.go:117] "RemoveContainer" containerID="d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8"
Sep 16 19:11:03.823244 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.823219    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:11:03.830858 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.830842    2579 scope.go:117] "RemoveContainer" containerID="d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981"
Sep 16 19:11:03.838325 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.838308    2579 scope.go:117] "RemoveContainer" containerID="d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8"
Sep 16 19:11:03.838600 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:11:03.838582    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8\": container with ID starting with d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8 not found: ID does not exist" containerID="d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8"
Sep 16 19:11:03.838649 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.838610    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8"} err="failed to get container status \"d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8\": rpc error: code = NotFound desc = could not find container \"d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8\": container with ID starting with d11c7a83d547ca821b2aae6ec78d21bd9ad8fba6a9db5b05f2cb5268bc0c3ee8 not found: ID does not exist"
Sep 16 19:11:03.838649 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.838626    2579 scope.go:117] "RemoveContainer" containerID="d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981"
Sep 16 19:11:03.838857 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:11:03.838841    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981\": container with ID starting with d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981 not found: ID does not exist" containerID="d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981"
Sep 16 19:11:03.838897 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.838860    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981"} err="failed to get container status \"d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981\": rpc error: code = NotFound desc = could not find container \"d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981\": container with ID starting with d15408b3488e9ed566ca4ca040e0df0b4ca96ba8975884ef45992851733d5981 not found: ID does not exist"
Sep 16 19:11:03.849547 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.849529    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"]
Sep 16 19:11:03.852530 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:03.852510    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4920-predictor-868cb5f55d-t4824"]
Sep 16 19:11:05.811830 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:05.811787    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" path="/var/lib/kubelet/pods/63e0cf4f-8afb-4e8c-81bf-0aad239c6876/volumes"
Sep 16 19:11:08.827690 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:08.827654    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:08.828247 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:08.828217    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:11:18.829097 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:18.829055    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:11:28.828858 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:28.828817    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:11:38.829082 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:38.829042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:11:48.829500 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:48.829472    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:11:49.456122 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.456086    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:49.591777 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.591733    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"]
Sep 16 19:11:49.591959 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.591934    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container" containerID="cri-o://40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a" gracePeriod=30
Sep 16 19:11:49.592068 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.591966    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kube-rbac-proxy" containerID="cri-o://3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447" gracePeriod=30
Sep 16 19:11:49.630520 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.630492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:49.740366 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.740309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:49.804591 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.804565    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"]
Sep 16 19:11:49.804942 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.804926    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kserve-container"
Sep 16 19:11:49.805013 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.804946    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kserve-container"
Sep 16 19:11:49.805013 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.804963    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kube-rbac-proxy"
Sep 16 19:11:49.805013 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.804969    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kube-rbac-proxy"
Sep 16 19:11:49.805111 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.805034    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kserve-container"
Sep 16 19:11:49.805111 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.805043    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e0cf4f-8afb-4e8c-81bf-0aad239c6876" containerName="kube-rbac-proxy"
Sep 16 19:11:49.808113 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.808092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:49.813330 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.813308    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d2d49-predictor-serving-cert\""
Sep 16 19:11:49.813748 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.813730    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d2d49-kube-rbac-proxy-sar-config\""
Sep 16 19:11:49.831398 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.831371    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"]
Sep 16 19:11:49.842728 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.842702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:49.874311 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.874286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5016495c-fa91-491c-a562-4f746e428fa6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:49.874423 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.874322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5016495c-fa91-491c-a562-4f746e428fa6-proxy-tls\") pod \"success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:49.874423 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.874346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjcd\" (UniqueName: \"kubernetes.io/projected/5016495c-fa91-491c-a562-4f746e428fa6-kube-api-access-rmjcd\") pod \"success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:49.975044 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.975015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5016495c-fa91-491c-a562-4f746e428fa6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:49.975198 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.975059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5016495c-fa91-491c-a562-4f746e428fa6-proxy-tls\") pod \"success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:49.975198 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.975077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjcd\" (UniqueName: \"kubernetes.io/projected/5016495c-fa91-491c-a562-4f746e428fa6-kube-api-access-rmjcd\") pod \"success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:49.975690 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.975666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5016495c-fa91-491c-a562-4f746e428fa6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:49.977652 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.977636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5016495c-fa91-491c-a562-4f746e428fa6-proxy-tls\") pod \"success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:49.994382 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:49.994330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjcd\" (UniqueName: \"kubernetes.io/projected/5016495c-fa91-491c-a562-4f746e428fa6-kube-api-access-rmjcd\") pod \"success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:50.016543 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:50.016517    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerID="3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447" exitCode=2
Sep 16 19:11:50.016626 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:50.016572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" event={"ID":"d2db5de4-8f0a-428f-b8ab-435f68d74cdc","Type":"ContainerDied","Data":"3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447"}
Sep 16 19:11:50.124052 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:50.124027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:50.431778 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:50.431750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"]
Sep 16 19:11:50.437040 ip-10-0-140-67 kubenswrapper[2579]: W0916 19:11:50.437011    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5016495c_fa91_491c_a562_4f746e428fa6.slice/crio-f8aecd1d54d0a6630ec59cbf7aae25b32124b5417268e9b07d05d3388929387b WatchSource:0}: Error finding container f8aecd1d54d0a6630ec59cbf7aae25b32124b5417268e9b07d05d3388929387b: Status 404 returned error can't find the container with id f8aecd1d54d0a6630ec59cbf7aae25b32124b5417268e9b07d05d3388929387b
Sep 16 19:11:50.587523 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:50.587488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:50.662665 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:50.662618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:51.022762 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:51.022726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" event={"ID":"5016495c-fa91-491c-a562-4f746e428fa6","Type":"ContainerStarted","Data":"129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f"}
Sep 16 19:11:51.023279 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:51.022798    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:51.023279 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:51.022827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" event={"ID":"5016495c-fa91-491c-a562-4f746e428fa6","Type":"ContainerStarted","Data":"3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728"}
Sep 16 19:11:51.023279 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:51.022858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" event={"ID":"5016495c-fa91-491c-a562-4f746e428fa6","Type":"ContainerStarted","Data":"f8aecd1d54d0a6630ec59cbf7aae25b32124b5417268e9b07d05d3388929387b"}
Sep 16 19:11:51.023279 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:51.022889    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:51.024143 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:51.024116    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:11:51.059175 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:51.059091    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" podStartSLOduration=2.0590501 podStartE2EDuration="2.0590501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:51.05646282 +0000 UTC m=+1865.721735283" watchObservedRunningTime="2026-09-16 19:11:51.0590501 +0000 UTC m=+1865.724322598"
Sep 16 19:11:51.670709 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:51.670666    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 16 19:11:51.675030 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:51.674994    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:11:52.027335 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.027236    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:11:52.735213 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.735188    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:11:52.802975 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.802944    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vfs8\" (UniqueName: \"kubernetes.io/projected/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-api-access-7vfs8\") pod \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") "
Sep 16 19:11:52.803183 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.803007    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-proxy-tls\") pod \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") "
Sep 16 19:11:52.803183 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.803050    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-rbac-proxy-sar-config\") pod \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\" (UID: \"d2db5de4-8f0a-428f-b8ab-435f68d74cdc\") "
Sep 16 19:11:52.803461 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.803437    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2db5de4-8f0a-428f-b8ab-435f68d74cdc" (UID: "d2db5de4-8f0a-428f-b8ab-435f68d74cdc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:52.805180 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.805143    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-api-access-7vfs8" (OuterVolumeSpecName: "kube-api-access-7vfs8") pod "d2db5de4-8f0a-428f-b8ab-435f68d74cdc" (UID: "d2db5de4-8f0a-428f-b8ab-435f68d74cdc"). InnerVolumeSpecName "kube-api-access-7vfs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:52.805255 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.805195    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2db5de4-8f0a-428f-b8ab-435f68d74cdc" (UID: "d2db5de4-8f0a-428f-b8ab-435f68d74cdc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:52.904297 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.904264    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vfs8\" (UniqueName: \"kubernetes.io/projected/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-api-access-7vfs8\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:11:52.904297 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.904288    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:11:52.904297 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:52.904299    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2db5de4-8f0a-428f-b8ab-435f68d74cdc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:11:53.033097 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.033064    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerID="40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a" exitCode=0
Sep 16 19:11:53.033551 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.033115    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"
Sep 16 19:11:53.033551 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.033174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" event={"ID":"d2db5de4-8f0a-428f-b8ab-435f68d74cdc","Type":"ContainerDied","Data":"40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a"}
Sep 16 19:11:53.033551 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.033250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg" event={"ID":"d2db5de4-8f0a-428f-b8ab-435f68d74cdc","Type":"ContainerDied","Data":"055e1e3766e16f14a10e990b422ae20999238faff7b974ba21a0b0dfcfbbd6d0"}
Sep 16 19:11:53.033551 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.033290    2579 scope.go:117] "RemoveContainer" containerID="3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447"
Sep 16 19:11:53.041997 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.041981    2579 scope.go:117] "RemoveContainer" containerID="40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a"
Sep 16 19:11:53.049686 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.049670    2579 scope.go:117] "RemoveContainer" containerID="3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447"
Sep 16 19:11:53.049953 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:11:53.049933    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447\": container with ID starting with 3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447 not found: ID does not exist" containerID="3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447"
Sep 16 19:11:53.050045 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.049959    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447"} err="failed to get container status \"3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447\": rpc error: code = NotFound desc = could not find container \"3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447\": container with ID starting with 3184c94c026c14d44bb0722285c804f1d8a447cd6d9175fedeb0cf3fcd701447 not found: ID does not exist"
Sep 16 19:11:53.050045 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.049975    2579 scope.go:117] "RemoveContainer" containerID="40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a"
Sep 16 19:11:53.050249 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:11:53.050232    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a\": container with ID starting with 40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a not found: ID does not exist" containerID="40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a"
Sep 16 19:11:53.050304 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.050253    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a"} err="failed to get container status \"40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a\": rpc error: code = NotFound desc = could not find container \"40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a\": container with ID starting with 40a00a6e53ef6b242d3503649c4c5f780e7e7b35f9c11d88377c906786b47c2a not found: ID does not exist"
Sep 16 19:11:53.061528 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.061508    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"]
Sep 16 19:11:53.070269 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.070246    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e3b29-predictor-7b4b4f756d-mzxqg"]
Sep 16 19:11:53.811876 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:53.811841    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" path="/var/lib/kubelet/pods/d2db5de4-8f0a-428f-b8ab-435f68d74cdc/volumes"
Sep 16 19:11:57.032624 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:57.032594    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:11:57.033223 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:11:57.033196    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:12:00.528083 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:00.528043    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:00.637747 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:00.637700    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:07.033407 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:07.033320    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:12:10.662763 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:10.662721    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:10.843029 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:10.842994    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"]
Sep 16 19:12:10.843253 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:10.843228    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container" containerID="cri-o://1cf494eb8b40d3c33dc53b00072ee4c72e965995ac118111e3f435b0e56a32b4" gracePeriod=30
Sep 16 19:12:10.843311 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:10.843258    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kube-rbac-proxy" containerID="cri-o://497d98c5337afa5264a9220432a1c2f1e641567f3322e3c4ef10d2b8d28c8732" gracePeriod=30
Sep 16 19:12:10.874639 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:10.874611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:10.990616 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:10.990588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:11.114992 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.114956    2579 generic.go:356] "Generic (PLEG): container finished" podID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerID="497d98c5337afa5264a9220432a1c2f1e641567f3322e3c4ef10d2b8d28c8732" exitCode=2
Sep 16 19:12:11.115181 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.115028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" event={"ID":"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3","Type":"ContainerDied","Data":"497d98c5337afa5264a9220432a1c2f1e641567f3322e3c4ef10d2b8d28c8732"}
Sep 16 19:12:11.134502 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.134469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:11.152867 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.152844    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"]
Sep 16 19:12:11.153276 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.153258    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container"
Sep 16 19:12:11.153356 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.153278    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container"
Sep 16 19:12:11.153356 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.153324    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kube-rbac-proxy"
Sep 16 19:12:11.153356 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.153333    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kube-rbac-proxy"
Sep 16 19:12:11.153515 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.153427    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kube-rbac-proxy"
Sep 16 19:12:11.153515 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.153443    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2db5de4-8f0a-428f-b8ab-435f68d74cdc" containerName="kserve-container"
Sep 16 19:12:11.156526 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.156509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.163301 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.163274    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0e2c5-predictor-serving-cert\""
Sep 16 19:12:11.163505 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.163483    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0e2c5-kube-rbac-proxy-sar-config\""
Sep 16 19:12:11.180578 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.180560    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"]
Sep 16 19:12:11.252146 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.252092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/619ee1fb-987e-405b-b5fb-2b947bb25a8e-proxy-tls\") pod \"success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") " pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.252146 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.252140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") " pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.252298 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.252226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r24x\" (UniqueName: \"kubernetes.io/projected/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-api-access-6r24x\") pod \"success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") " pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.353346 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.353319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") " pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.353473 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.353364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r24x\" (UniqueName: \"kubernetes.io/projected/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-api-access-6r24x\") pod \"success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") " pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.353473 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.353409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/619ee1fb-987e-405b-b5fb-2b947bb25a8e-proxy-tls\") pod \"success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") " pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.354411 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.354383    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") " pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.356061 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.356030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/619ee1fb-987e-405b-b5fb-2b947bb25a8e-proxy-tls\") pod \"success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") " pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.366491 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.366470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r24x\" (UniqueName: \"kubernetes.io/projected/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-api-access-6r24x\") pod \"success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") " pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.466732 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.466708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:11.695096 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.695058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:11.754983 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.754961    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"]
Sep 16 19:12:11.757008 ip-10-0-140-67 kubenswrapper[2579]: W0916 19:12:11.756978    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619ee1fb_987e_405b_b5fb_2b947bb25a8e.slice/crio-1431755409df0b400034e3515687be17f2f4c282aed0923d1d2002b704bc8cd6 WatchSource:0}: Error finding container 1431755409df0b400034e3515687be17f2f4c282aed0923d1d2002b704bc8cd6: Status 404 returned error can't find the container with id 1431755409df0b400034e3515687be17f2f4c282aed0923d1d2002b704bc8cd6
Sep 16 19:12:11.839190 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:11.839165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:12.121461 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:12.121429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" event={"ID":"619ee1fb-987e-405b-b5fb-2b947bb25a8e","Type":"ContainerStarted","Data":"2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082"}
Sep 16 19:12:12.121640 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:12.121490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" event={"ID":"619ee1fb-987e-405b-b5fb-2b947bb25a8e","Type":"ContainerStarted","Data":"2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0"}
Sep 16 19:12:12.121640 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:12.121512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" event={"ID":"619ee1fb-987e-405b-b5fb-2b947bb25a8e","Type":"ContainerStarted","Data":"1431755409df0b400034e3515687be17f2f4c282aed0923d1d2002b704bc8cd6"}
Sep 16 19:12:12.121640 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:12.121550    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:12.121640 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:12.121584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:12.122980 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:12.122950    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:12:12.151925 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:12.151880    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podStartSLOduration=1.151865006 podStartE2EDuration="1.151865006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:12.149235809 +0000 UTC m=+1886.814508258" watchObservedRunningTime="2026-09-16 19:12:12.151865006 +0000 UTC m=+1886.817137437"
Sep 16 19:12:13.125779 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:13.125741    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:12:13.823644 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:13.823605    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:12:14.132335 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.132301    2579 generic.go:356] "Generic (PLEG): container finished" podID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerID="1cf494eb8b40d3c33dc53b00072ee4c72e965995ac118111e3f435b0e56a32b4" exitCode=0
Sep 16 19:12:14.132666 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.132375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" event={"ID":"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3","Type":"ContainerDied","Data":"1cf494eb8b40d3c33dc53b00072ee4c72e965995ac118111e3f435b0e56a32b4"}
Sep 16 19:12:14.188963 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.188940    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:12:14.281975 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.281943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-proxy-tls\") pod \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") "
Sep 16 19:12:14.281975 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.281979    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpj24\" (UniqueName: \"kubernetes.io/projected/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-api-access-cpj24\") pod \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") "
Sep 16 19:12:14.282851 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.282081    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-rbac-proxy-sar-config\") pod \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\" (UID: \"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3\") "
Sep 16 19:12:14.282851 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.282538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" (UID: "f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:14.284904 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.284877    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" (UID: "f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:14.285035 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.284919    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-api-access-cpj24" (OuterVolumeSpecName: "kube-api-access-cpj24") pod "f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" (UID: "f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3"). InnerVolumeSpecName "kube-api-access-cpj24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:14.383534 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.383450    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpj24\" (UniqueName: \"kubernetes.io/projected/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-api-access-cpj24\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:12:14.383534 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.383477    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:12:14.383534 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:14.383487    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:12:15.138868 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:15.138842    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"
Sep 16 19:12:15.139332 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:15.138867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7" event={"ID":"f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3","Type":"ContainerDied","Data":"cf23ba6d9a0e36d215cb4c3742b42bcab7a94b3239a0d8f8a0dc67ffb283ea5e"}
Sep 16 19:12:15.139332 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:15.138944    2579 scope.go:117] "RemoveContainer" containerID="497d98c5337afa5264a9220432a1c2f1e641567f3322e3c4ef10d2b8d28c8732"
Sep 16 19:12:15.147733 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:15.147707    2579 scope.go:117] "RemoveContainer" containerID="1cf494eb8b40d3c33dc53b00072ee4c72e965995ac118111e3f435b0e56a32b4"
Sep 16 19:12:15.165291 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:15.165271    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"]
Sep 16 19:12:15.175787 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:15.175767    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-617e5-predictor-776d98c8-5r9t7"]
Sep 16 19:12:15.811377 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:15.811350    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" path="/var/lib/kubelet/pods/f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3/volumes"
Sep 16 19:12:17.033333 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:17.033293    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:12:18.130742 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:18.130711    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:12:18.131114 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:18.131088    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:12:27.033956 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:27.033913    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:12:28.131536 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:28.131492    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:12:37.033889 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:37.033853    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:12:38.131313 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:38.131271    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:12:48.131336 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:48.131288    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:12:49.872505 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:49.872471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:50.132075 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:50.131989    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:58.132515 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:12:58.132480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:13:11.196350 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:13:11.196314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:11.303321 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:13:11.303288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:57.138658 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:15:57.138627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:07.867577 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:16:07.867545    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:13.196044 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:17:13.196012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:17.205212 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:17:17.205143    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:23.374683 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:17:23.374645    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:23.464465 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:17:23.464429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:01.201515 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:20:01.201402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:25.720235 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:25.720196    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:25.830733 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:25.830697    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"]
Sep 16 19:21:25.830942 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:25.830895    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container" containerID="cri-o://2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0" gracePeriod=30
Sep 16 19:21:25.831001 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:25.830929    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kube-rbac-proxy" containerID="cri-o://2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082" gracePeriod=30
Sep 16 19:21:26.434695 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:26.434656    2579 generic.go:356] "Generic (PLEG): container finished" podID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerID="2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082" exitCode=2
Sep 16 19:21:26.434858 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:26.434729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" event={"ID":"619ee1fb-987e-405b-b5fb-2b947bb25a8e","Type":"ContainerDied","Data":"2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082"}
Sep 16 19:21:26.818441 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:26.818361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:28.126850 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.126809    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 19:21:28.131187 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.131139    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:21:28.578744 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.578723    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:21:28.658199 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.658179    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r24x\" (UniqueName: \"kubernetes.io/projected/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-api-access-6r24x\") pod \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") "
Sep 16 19:21:28.658331 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.658222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-rbac-proxy-sar-config\") pod \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") "
Sep 16 19:21:28.658331 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.658267    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/619ee1fb-987e-405b-b5fb-2b947bb25a8e-proxy-tls\") pod \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\" (UID: \"619ee1fb-987e-405b-b5fb-2b947bb25a8e\") "
Sep 16 19:21:28.658649 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.658620    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "619ee1fb-987e-405b-b5fb-2b947bb25a8e" (UID: "619ee1fb-987e-405b-b5fb-2b947bb25a8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:28.660452 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.660427    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/619ee1fb-987e-405b-b5fb-2b947bb25a8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "619ee1fb-987e-405b-b5fb-2b947bb25a8e" (UID: "619ee1fb-987e-405b-b5fb-2b947bb25a8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:28.660555 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.660488    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-api-access-6r24x" (OuterVolumeSpecName: "kube-api-access-6r24x") pod "619ee1fb-987e-405b-b5fb-2b947bb25a8e" (UID: "619ee1fb-987e-405b-b5fb-2b947bb25a8e"). InnerVolumeSpecName "kube-api-access-6r24x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:28.759047 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.758997    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r24x\" (UniqueName: \"kubernetes.io/projected/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-api-access-6r24x\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:21:28.759047 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.759017    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/619ee1fb-987e-405b-b5fb-2b947bb25a8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:21:28.759047 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:28.759026    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/619ee1fb-987e-405b-b5fb-2b947bb25a8e-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:21:29.449452 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.449420    2579 generic.go:356] "Generic (PLEG): container finished" podID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerID="2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0" exitCode=0
Sep 16 19:21:29.449825 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.449476    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"
Sep 16 19:21:29.449825 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.449481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" event={"ID":"619ee1fb-987e-405b-b5fb-2b947bb25a8e","Type":"ContainerDied","Data":"2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0"}
Sep 16 19:21:29.449825 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.449530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd" event={"ID":"619ee1fb-987e-405b-b5fb-2b947bb25a8e","Type":"ContainerDied","Data":"1431755409df0b400034e3515687be17f2f4c282aed0923d1d2002b704bc8cd6"}
Sep 16 19:21:29.449825 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.449555    2579 scope.go:117] "RemoveContainer" containerID="2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082"
Sep 16 19:21:29.458436 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.458417    2579 scope.go:117] "RemoveContainer" containerID="2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0"
Sep 16 19:21:29.465838 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.465819    2579 scope.go:117] "RemoveContainer" containerID="2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082"
Sep 16 19:21:29.466081 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:21:29.466066    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082\": container with ID starting with 2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082 not found: ID does not exist" containerID="2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082"
Sep 16 19:21:29.466131 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.466089    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082"} err="failed to get container status \"2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082\": rpc error: code = NotFound desc = could not find container \"2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082\": container with ID starting with 2d25e5e940a885f0485611d3de0d7403d01cca221a2845a5aedf76b784071082 not found: ID does not exist"
Sep 16 19:21:29.466131 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.466102    2579 scope.go:117] "RemoveContainer" containerID="2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0"
Sep 16 19:21:29.466375 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:21:29.466357    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0\": container with ID starting with 2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0 not found: ID does not exist" containerID="2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0"
Sep 16 19:21:29.466435 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.466385    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0"} err="failed to get container status \"2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0\": rpc error: code = NotFound desc = could not find container \"2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0\": container with ID starting with 2487e8bb162bce08e65011ee44b8c10e5b3a019ac61a859dd203ded3075ccaa0 not found: ID does not exist"
Sep 16 19:21:29.486501 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.486474    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"]
Sep 16 19:21:29.499621 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.499598    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0e2c5-predictor-7956b5b96-jwqzd"]
Sep 16 19:21:29.812981 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:21:29.812908    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" path="/var/lib/kubelet/pods/619ee1fb-987e-405b-b5fb-2b947bb25a8e/volumes"
Sep 16 19:23:01.507797 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:23:01.507723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:50.172083 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:23:50.172047    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:53.646379 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:28:53.646304    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:09.149022 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:09.148987    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:09.269962 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:09.269925    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"]
Sep 16 19:29:09.270189 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:09.270143    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kserve-container" containerID="cri-o://3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728" gracePeriod=30
Sep 16 19:29:09.270322 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:09.270207    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kube-rbac-proxy" containerID="cri-o://129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f" gracePeriod=30
Sep 16 19:29:09.318799 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:09.318761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:09.393356 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:09.393319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:10.347359 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:10.347325    2579 generic.go:356] "Generic (PLEG): container finished" podID="5016495c-fa91-491c-a562-4f746e428fa6" containerID="129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f" exitCode=2
Sep 16 19:29:10.347725 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:10.347381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" event={"ID":"5016495c-fa91-491c-a562-4f746e428fa6","Type":"ContainerDied","Data":"129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f"}
Sep 16 19:29:11.039815 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.039780    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w8dgx/must-gather-xwbkh"]
Sep 16 19:29:11.040201 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040186    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kube-rbac-proxy"
Sep 16 19:29:11.040253 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040204    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kube-rbac-proxy"
Sep 16 19:29:11.040253 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040216    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container"
Sep 16 19:29:11.040253 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040222    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container"
Sep 16 19:29:11.040253 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040229    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kube-rbac-proxy"
Sep 16 19:29:11.040253 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040235    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kube-rbac-proxy"
Sep 16 19:29:11.040253 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040248    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container"
Sep 16 19:29:11.040253 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040253    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container"
Sep 16 19:29:11.040466 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040301    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kserve-container"
Sep 16 19:29:11.040466 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040309    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kserve-container"
Sep 16 19:29:11.040466 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040316    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f903b59d-cf5a-496c-9bd3-1ca24dd9f7d3" containerName="kube-rbac-proxy"
Sep 16 19:29:11.040466 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.040322    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="619ee1fb-987e-405b-b5fb-2b947bb25a8e" containerName="kube-rbac-proxy"
Sep 16 19:29:11.042926 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.042904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:11.045753 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.045734    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w8dgx\"/\"kube-root-ca.crt\""
Sep 16 19:29:11.046819 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.046803    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w8dgx\"/\"default-dockercfg-rg9kp\""
Sep 16 19:29:11.046887 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.046803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w8dgx\"/\"openshift-service-ca.crt\""
Sep 16 19:29:11.053864 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.053845    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w8dgx/must-gather-xwbkh"]
Sep 16 19:29:11.056136 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.056117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:11.116045 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.116015    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:11.133206 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.133183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drs8d\" (UniqueName: \"kubernetes.io/projected/746c0bf3-4784-48e5-aa97-20f2aba0341c-kube-api-access-drs8d\") pod \"must-gather-xwbkh\" (UID: \"746c0bf3-4784-48e5-aa97-20f2aba0341c\") " pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:11.133296 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.133221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/746c0bf3-4784-48e5-aa97-20f2aba0341c-must-gather-output\") pod \"must-gather-xwbkh\" (UID: \"746c0bf3-4784-48e5-aa97-20f2aba0341c\") " pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:11.234581 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.234542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/746c0bf3-4784-48e5-aa97-20f2aba0341c-must-gather-output\") pod \"must-gather-xwbkh\" (UID: \"746c0bf3-4784-48e5-aa97-20f2aba0341c\") " pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:11.234690 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.234648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drs8d\" (UniqueName: \"kubernetes.io/projected/746c0bf3-4784-48e5-aa97-20f2aba0341c-kube-api-access-drs8d\") pod \"must-gather-xwbkh\" (UID: \"746c0bf3-4784-48e5-aa97-20f2aba0341c\") " pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:11.234872 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.234856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/746c0bf3-4784-48e5-aa97-20f2aba0341c-must-gather-output\") pod \"must-gather-xwbkh\" (UID: \"746c0bf3-4784-48e5-aa97-20f2aba0341c\") " pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:11.247636 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.247608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drs8d\" (UniqueName: \"kubernetes.io/projected/746c0bf3-4784-48e5-aa97-20f2aba0341c-kube-api-access-drs8d\") pod \"must-gather-xwbkh\" (UID: \"746c0bf3-4784-48e5-aa97-20f2aba0341c\") " pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:11.367753 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.367727    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:11.643655 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.643558    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w8dgx/must-gather-xwbkh"]
Sep 16 19:29:11.646520 ip-10-0-140-67 kubenswrapper[2579]: W0916 19:29:11.646480    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod746c0bf3_4784_48e5_aa97_20f2aba0341c.slice/crio-9543f45ef3729c289bc1b65aad3d6fb42683f6926e0bbf3d6f2a4e73d984b6cc WatchSource:0}: Error finding container 9543f45ef3729c289bc1b65aad3d6fb42683f6926e0bbf3d6f2a4e73d984b6cc: Status 404 returned error can't find the container with id 9543f45ef3729c289bc1b65aad3d6fb42683f6926e0bbf3d6f2a4e73d984b6cc
Sep 16 19:29:11.648676 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:11.648661    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:29:12.027810 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.027714    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 16 19:29:12.208134 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.208108    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:29:12.342327 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.342298    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5016495c-fa91-491c-a562-4f746e428fa6-proxy-tls\") pod \"5016495c-fa91-491c-a562-4f746e428fa6\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") "
Sep 16 19:29:12.342488 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.342398    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5016495c-fa91-491c-a562-4f746e428fa6-kube-rbac-proxy-sar-config\") pod \"5016495c-fa91-491c-a562-4f746e428fa6\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") "
Sep 16 19:29:12.342488 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.342435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmjcd\" (UniqueName: \"kubernetes.io/projected/5016495c-fa91-491c-a562-4f746e428fa6-kube-api-access-rmjcd\") pod \"5016495c-fa91-491c-a562-4f746e428fa6\" (UID: \"5016495c-fa91-491c-a562-4f746e428fa6\") "
Sep 16 19:29:12.343127 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.343086    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5016495c-fa91-491c-a562-4f746e428fa6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5016495c-fa91-491c-a562-4f746e428fa6" (UID: "5016495c-fa91-491c-a562-4f746e428fa6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:12.349739 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.349710    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5016495c-fa91-491c-a562-4f746e428fa6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5016495c-fa91-491c-a562-4f746e428fa6" (UID: "5016495c-fa91-491c-a562-4f746e428fa6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:12.356711 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.356682    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5016495c-fa91-491c-a562-4f746e428fa6-kube-api-access-rmjcd" (OuterVolumeSpecName: "kube-api-access-rmjcd") pod "5016495c-fa91-491c-a562-4f746e428fa6" (UID: "5016495c-fa91-491c-a562-4f746e428fa6"). InnerVolumeSpecName "kube-api-access-rmjcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:12.362930 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.361873    2579 generic.go:356] "Generic (PLEG): container finished" podID="5016495c-fa91-491c-a562-4f746e428fa6" containerID="3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728" exitCode=0
Sep 16 19:29:12.362930 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.362040    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"
Sep 16 19:29:12.362930 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.362836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" event={"ID":"5016495c-fa91-491c-a562-4f746e428fa6","Type":"ContainerDied","Data":"3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728"}
Sep 16 19:29:12.362930 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.362923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk" event={"ID":"5016495c-fa91-491c-a562-4f746e428fa6","Type":"ContainerDied","Data":"f8aecd1d54d0a6630ec59cbf7aae25b32124b5417268e9b07d05d3388929387b"}
Sep 16 19:29:12.363212 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.362981    2579 scope.go:117] "RemoveContainer" containerID="129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f"
Sep 16 19:29:12.368798 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.368756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w8dgx/must-gather-xwbkh" event={"ID":"746c0bf3-4784-48e5-aa97-20f2aba0341c","Type":"ContainerStarted","Data":"9543f45ef3729c289bc1b65aad3d6fb42683f6926e0bbf3d6f2a4e73d984b6cc"}
Sep 16 19:29:12.379194 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.378974    2579 scope.go:117] "RemoveContainer" containerID="3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728"
Sep 16 19:29:12.393060 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.392974    2579 scope.go:117] "RemoveContainer" containerID="129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f"
Sep 16 19:29:12.393740 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:29:12.393599    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f\": container with ID starting with 129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f not found: ID does not exist" containerID="129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f"
Sep 16 19:29:12.393740 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.393637    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f"} err="failed to get container status \"129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f\": rpc error: code = NotFound desc = could not find container \"129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f\": container with ID starting with 129ed4e28d5c438602cc1ff1fd15171f799d612079f1c4b57cd2108435428b2f not found: ID does not exist"
Sep 16 19:29:12.393740 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.393659    2579 scope.go:117] "RemoveContainer" containerID="3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728"
Sep 16 19:29:12.394669 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:29:12.394600    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728\": container with ID starting with 3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728 not found: ID does not exist" containerID="3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728"
Sep 16 19:29:12.394669 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.394630    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728"} err="failed to get container status \"3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728\": rpc error: code = NotFound desc = could not find container \"3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728\": container with ID starting with 3c3421d3d8c88b48d1987b7e660876b83c0bb1df2883c823c8df44df6d142728 not found: ID does not exist"
Sep 16 19:29:12.404619 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.404599    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"]
Sep 16 19:29:12.409991 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.409282    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d49-predictor-c9cf5cd84-q4dlk"]
Sep 16 19:29:12.443926 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.443902    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5016495c-fa91-491c-a562-4f746e428fa6-proxy-tls\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:29:12.444044 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.443934    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5016495c-fa91-491c-a562-4f746e428fa6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:29:12.444044 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:12.443950    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjcd\" (UniqueName: \"kubernetes.io/projected/5016495c-fa91-491c-a562-4f746e428fa6-kube-api-access-rmjcd\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:29:13.813408 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:13.813379    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5016495c-fa91-491c-a562-4f746e428fa6" path="/var/lib/kubelet/pods/5016495c-fa91-491c-a562-4f746e428fa6/volumes"
Sep 16 19:29:18.400023 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:18.399987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w8dgx/must-gather-xwbkh" event={"ID":"746c0bf3-4784-48e5-aa97-20f2aba0341c","Type":"ContainerStarted","Data":"806efcec10216b1c3d2dedc0583bbd3993d1b07acf6ff67b7b91e64cb3299405"}
Sep 16 19:29:18.400023 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:18.400031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w8dgx/must-gather-xwbkh" event={"ID":"746c0bf3-4784-48e5-aa97-20f2aba0341c","Type":"ContainerStarted","Data":"fcf13813646fda9e92a23ce94837ee96d9de3c747b94d3420608fc2dc58d719a"}
Sep 16 19:29:18.422484 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:18.422432    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w8dgx/must-gather-xwbkh" podStartSLOduration=1.298238786 podStartE2EDuration="7.422417074s" totalImagesPullingTime="6.124178288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:18.421492305 +0000 UTC m=+2913.086764736" watchObservedRunningTime="2026-09-16 19:29:18.422417074 +0000 UTC m=+2913.087689504"
Sep 16 19:29:40.507265 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:40.507226    2579 generic.go:356] "Generic (PLEG): container finished" podID="746c0bf3-4784-48e5-aa97-20f2aba0341c" containerID="fcf13813646fda9e92a23ce94837ee96d9de3c747b94d3420608fc2dc58d719a" exitCode=0
Sep 16 19:29:40.507683 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:40.507300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w8dgx/must-gather-xwbkh" event={"ID":"746c0bf3-4784-48e5-aa97-20f2aba0341c","Type":"ContainerDied","Data":"fcf13813646fda9e92a23ce94837ee96d9de3c747b94d3420608fc2dc58d719a"}
Sep 16 19:29:40.507683 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:40.507621    2579 scope.go:117] "RemoveContainer" containerID="fcf13813646fda9e92a23ce94837ee96d9de3c747b94d3420608fc2dc58d719a"
Sep 16 19:29:40.709526 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:40.709493    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-w8dgx_must-gather-xwbkh_746c0bf3-4784-48e5-aa97-20f2aba0341c/gather/0.log"
Sep 16 19:29:41.220615 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.220524    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:41.390118 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.390085    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sc8ck/must-gather-lfttc"]
Sep 16 19:29:41.390451 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.390438    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kube-rbac-proxy"
Sep 16 19:29:41.390513 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.390453    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kube-rbac-proxy"
Sep 16 19:29:41.390513 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.390472    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kserve-container"
Sep 16 19:29:41.390513 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.390477    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kserve-container"
Sep 16 19:29:41.390633 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.390530    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kserve-container"
Sep 16 19:29:41.390633 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.390541    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5016495c-fa91-491c-a562-4f746e428fa6" containerName="kube-rbac-proxy"
Sep 16 19:29:41.392647 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.392626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sc8ck/must-gather-lfttc"
Sep 16 19:29:41.395271 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.395249    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sc8ck\"/\"default-dockercfg-zs942\""
Sep 16 19:29:41.396281 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.396259    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sc8ck\"/\"openshift-service-ca.crt\""
Sep 16 19:29:41.396358 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.396279    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sc8ck\"/\"kube-root-ca.crt\""
Sep 16 19:29:41.398268 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.398238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:41.404366 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.404344    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sc8ck/must-gather-lfttc"]
Sep 16 19:29:41.404628 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.404610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/66d8d92d-3825-491f-8d5c-e2ac5e56fd39-must-gather-output\") pod \"must-gather-lfttc\" (UID: \"66d8d92d-3825-491f-8d5c-e2ac5e56fd39\") " pod="openshift-must-gather-sc8ck/must-gather-lfttc"
Sep 16 19:29:41.404686 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.404668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ml7b\" (UniqueName: \"kubernetes.io/projected/66d8d92d-3825-491f-8d5c-e2ac5e56fd39-kube-api-access-2ml7b\") pod \"must-gather-lfttc\" (UID: \"66d8d92d-3825-491f-8d5c-e2ac5e56fd39\") " pod="openshift-must-gather-sc8ck/must-gather-lfttc"
Sep 16 19:29:41.494790 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.494729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:41.505011 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.504991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml7b\" (UniqueName: \"kubernetes.io/projected/66d8d92d-3825-491f-8d5c-e2ac5e56fd39-kube-api-access-2ml7b\") pod \"must-gather-lfttc\" (UID: \"66d8d92d-3825-491f-8d5c-e2ac5e56fd39\") " pod="openshift-must-gather-sc8ck/must-gather-lfttc"
Sep 16 19:29:41.505080 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.505029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/66d8d92d-3825-491f-8d5c-e2ac5e56fd39-must-gather-output\") pod \"must-gather-lfttc\" (UID: \"66d8d92d-3825-491f-8d5c-e2ac5e56fd39\") " pod="openshift-must-gather-sc8ck/must-gather-lfttc"
Sep 16 19:29:41.505359 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.505344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/66d8d92d-3825-491f-8d5c-e2ac5e56fd39-must-gather-output\") pod \"must-gather-lfttc\" (UID: \"66d8d92d-3825-491f-8d5c-e2ac5e56fd39\") " pod="openshift-must-gather-sc8ck/must-gather-lfttc"
Sep 16 19:29:41.516799 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.516775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ml7b\" (UniqueName: \"kubernetes.io/projected/66d8d92d-3825-491f-8d5c-e2ac5e56fd39-kube-api-access-2ml7b\") pod \"must-gather-lfttc\" (UID: \"66d8d92d-3825-491f-8d5c-e2ac5e56fd39\") " pod="openshift-must-gather-sc8ck/must-gather-lfttc"
Sep 16 19:29:41.702639 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.702616    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sc8ck/must-gather-lfttc"
Sep 16 19:29:41.973841 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:41.973799    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sc8ck/must-gather-lfttc"]
Sep 16 19:29:41.977674 ip-10-0-140-67 kubenswrapper[2579]: W0916 19:29:41.977650    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66d8d92d_3825_491f_8d5c_e2ac5e56fd39.slice/crio-422921a950c0f763a18004e36fb609c6c72cf273450827c49e567c81def77d9c WatchSource:0}: Error finding container 422921a950c0f763a18004e36fb609c6c72cf273450827c49e567c81def77d9c: Status 404 returned error can't find the container with id 422921a950c0f763a18004e36fb609c6c72cf273450827c49e567c81def77d9c
Sep 16 19:29:42.518689 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:42.518657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sc8ck/must-gather-lfttc" event={"ID":"66d8d92d-3825-491f-8d5c-e2ac5e56fd39","Type":"ContainerStarted","Data":"422921a950c0f763a18004e36fb609c6c72cf273450827c49e567c81def77d9c"}
Sep 16 19:29:43.525469 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:43.525434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sc8ck/must-gather-lfttc" event={"ID":"66d8d92d-3825-491f-8d5c-e2ac5e56fd39","Type":"ContainerStarted","Data":"2e52c03a7947e47b0da1f570cb1e69aac2189678821fede675564c24f964e10e"}
Sep 16 19:29:43.525469 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:43.525474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sc8ck/must-gather-lfttc" event={"ID":"66d8d92d-3825-491f-8d5c-e2ac5e56fd39","Type":"ContainerStarted","Data":"e0a2c32bfbfbe71fa7b1826bb42030af8b5fe14df11b336a977dffffbd069156"}
Sep 16 19:29:43.550993 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:43.550933    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sc8ck/must-gather-lfttc" podStartSLOduration=1.848005574 podStartE2EDuration="2.550914223s" totalImagesPullingTime="702.908649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:43.55040929 +0000 UTC m=+2938.215681721" watchObservedRunningTime="2026-09-16 19:29:43.550914223 +0000 UTC m=+2938.216186655"
Sep 16 19:29:44.252423 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:44.252388    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-46l2m_5b52d6a7-170d-47a1-b771-ff55013146d6/global-pull-secret-syncer/0.log"
Sep 16 19:29:44.396608 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:44.396578    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log"
Sep 16 19:29:44.554951 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:44.554872    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-67.ec2.internal_a79a00728263f7068c7da6157868c572/haproxy/0.log"
Sep 16 19:29:46.197435 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.197379    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-w8dgx/must-gather-xwbkh"]
Sep 16 19:29:46.197936 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.197665    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-w8dgx/must-gather-xwbkh" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" containerName="copy" containerID="cri-o://806efcec10216b1c3d2dedc0583bbd3993d1b07acf6ff67b7b91e64cb3299405" gracePeriod=2
Sep 16 19:29:46.204521 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.204490    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-w8dgx/must-gather-xwbkh"]
Sep 16 19:29:46.547208 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.547182    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-w8dgx_must-gather-xwbkh_746c0bf3-4784-48e5-aa97-20f2aba0341c/copy/0.log"
Sep 16 19:29:46.547887 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.547864    2579 generic.go:356] "Generic (PLEG): container finished" podID="746c0bf3-4784-48e5-aa97-20f2aba0341c" containerID="806efcec10216b1c3d2dedc0583bbd3993d1b07acf6ff67b7b91e64cb3299405" exitCode=143
Sep 16 19:29:46.632083 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.632053    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-w8dgx_must-gather-xwbkh_746c0bf3-4784-48e5-aa97-20f2aba0341c/copy/0.log"
Sep 16 19:29:46.635369 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.635347    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:46.641173 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:29:46.637869    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xwbkh\" is forbidden: User \"system:node:ip-10-0-140-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-w8dgx\": no relationship found between node 'ip-10-0-140-67.ec2.internal' and this object" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:46.661648 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.661613    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drs8d\" (UniqueName: \"kubernetes.io/projected/746c0bf3-4784-48e5-aa97-20f2aba0341c-kube-api-access-drs8d\") pod \"746c0bf3-4784-48e5-aa97-20f2aba0341c\" (UID: \"746c0bf3-4784-48e5-aa97-20f2aba0341c\") "
Sep 16 19:29:46.661789 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.661758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/746c0bf3-4784-48e5-aa97-20f2aba0341c-must-gather-output\") pod \"746c0bf3-4784-48e5-aa97-20f2aba0341c\" (UID: \"746c0bf3-4784-48e5-aa97-20f2aba0341c\") "
Sep 16 19:29:46.665326 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.665288    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746c0bf3-4784-48e5-aa97-20f2aba0341c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "746c0bf3-4784-48e5-aa97-20f2aba0341c" (UID: "746c0bf3-4784-48e5-aa97-20f2aba0341c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:46.666116 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.666085    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746c0bf3-4784-48e5-aa97-20f2aba0341c-kube-api-access-drs8d" (OuterVolumeSpecName: "kube-api-access-drs8d") pod "746c0bf3-4784-48e5-aa97-20f2aba0341c" (UID: "746c0bf3-4784-48e5-aa97-20f2aba0341c"). InnerVolumeSpecName "kube-api-access-drs8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:46.763112 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.763064    2579 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/746c0bf3-4784-48e5-aa97-20f2aba0341c-must-gather-output\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:29:46.763112 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:46.763112    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drs8d\" (UniqueName: \"kubernetes.io/projected/746c0bf3-4784-48e5-aa97-20f2aba0341c-kube-api-access-drs8d\") on node \"ip-10-0-140-67.ec2.internal\" DevicePath \"\""
Sep 16 19:29:47.339788 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:47.339742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:47.428672 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:47.428603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spz42_4ea0010d-1f3d-40ee-8b69-787ecf68bf84/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:47.557710 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:47.556371    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-w8dgx_must-gather-xwbkh_746c0bf3-4784-48e5-aa97-20f2aba0341c/copy/0.log"
Sep 16 19:29:47.557710 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:47.556824    2579 scope.go:117] "RemoveContainer" containerID="806efcec10216b1c3d2dedc0583bbd3993d1b07acf6ff67b7b91e64cb3299405"
Sep 16 19:29:47.557710 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:47.556983    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:47.571397 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:29:47.571350    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xwbkh\" is forbidden: User \"system:node:ip-10-0-140-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-w8dgx\": no relationship found between node 'ip-10-0-140-67.ec2.internal' and this object" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:47.577634 ip-10-0-140-67 kubenswrapper[2579]: E0916 19:29:47.577583    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xwbkh\" is forbidden: User \"system:node:ip-10-0-140-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-w8dgx\": no relationship found between node 'ip-10-0-140-67.ec2.internal' and this object" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" pod="openshift-must-gather-w8dgx/must-gather-xwbkh"
Sep 16 19:29:47.581954 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:47.581619    2579 scope.go:117] "RemoveContainer" containerID="fcf13813646fda9e92a23ce94837ee96d9de3c747b94d3420608fc2dc58d719a"
Sep 16 19:29:47.814406 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:47.814369    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" path="/var/lib/kubelet/pods/746c0bf3-4784-48e5-aa97-20f2aba0341c/volumes"
Sep 16 19:29:48.602106 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.602075    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5cafeaa-779d-4c84-a719-8af92f10e364/alertmanager/0.log"
Sep 16 19:29:48.636427 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.636317    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5cafeaa-779d-4c84-a719-8af92f10e364/config-reloader/0.log"
Sep 16 19:29:48.663895 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.663858    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5cafeaa-779d-4c84-a719-8af92f10e364/kube-rbac-proxy-web/0.log"
Sep 16 19:29:48.694518 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.694464    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5cafeaa-779d-4c84-a719-8af92f10e364/kube-rbac-proxy/0.log"
Sep 16 19:29:48.724905 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.724877    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5cafeaa-779d-4c84-a719-8af92f10e364/kube-rbac-proxy-metric/0.log"
Sep 16 19:29:48.754427 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.754390    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5cafeaa-779d-4c84-a719-8af92f10e364/prom-label-proxy/0.log"
Sep 16 19:29:48.789643 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.789614    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5cafeaa-779d-4c84-a719-8af92f10e364/init-config-reloader/0.log"
Sep 16 19:29:48.868545 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.868516    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bj85p_c5aba59c-7e57-4c03-9449-abd9dd2e40ba/kube-state-metrics/0.log"
Sep 16 19:29:48.901187 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.901090    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bj85p_c5aba59c-7e57-4c03-9449-abd9dd2e40ba/kube-rbac-proxy-main/0.log"
Sep 16 19:29:48.933905 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.933870    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bj85p_c5aba59c-7e57-4c03-9449-abd9dd2e40ba/kube-rbac-proxy-self/0.log"
Sep 16 19:29:48.980122 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:48.980099    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64644594b-5c8cb_b651e8e0-92d6-40b2-8064-7523c2ca41a8/metrics-server/0.log"
Sep 16 19:29:49.010624 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.010598    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-59pp5_8989cb34-2530-415a-bb8a-1a3542e6cfff/monitoring-plugin/0.log"
Sep 16 19:29:49.045942 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.045919    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hwz99_03e62226-e882-4e47-8f9f-d98e180d3205/node-exporter/0.log"
Sep 16 19:29:49.075922 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.075896    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hwz99_03e62226-e882-4e47-8f9f-d98e180d3205/kube-rbac-proxy/0.log"
Sep 16 19:29:49.107609 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.107582    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hwz99_03e62226-e882-4e47-8f9f-d98e180d3205/init-textfile/0.log"
Sep 16 19:29:49.372754 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.372720    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mm67w_4cfc7333-89d1-44f5-9d85-27547bcd574c/kube-rbac-proxy-main/0.log"
Sep 16 19:29:49.408775 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.408749    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mm67w_4cfc7333-89d1-44f5-9d85-27547bcd574c/kube-rbac-proxy-self/0.log"
Sep 16 19:29:49.439784 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.439756    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mm67w_4cfc7333-89d1-44f5-9d85-27547bcd574c/openshift-state-metrics/0.log"
Sep 16 19:29:49.486791 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.486757    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd3074b6-6505-4bd3-8738-213aa47b12c1/prometheus/0.log"
Sep 16 19:29:49.517589 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.517562    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd3074b6-6505-4bd3-8738-213aa47b12c1/config-reloader/0.log"
Sep 16 19:29:49.546474 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.546429    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd3074b6-6505-4bd3-8738-213aa47b12c1/thanos-sidecar/0.log"
Sep 16 19:29:49.588298 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.588234    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd3074b6-6505-4bd3-8738-213aa47b12c1/kube-rbac-proxy-web/0.log"
Sep 16 19:29:49.622262 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.622237    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd3074b6-6505-4bd3-8738-213aa47b12c1/kube-rbac-proxy/0.log"
Sep 16 19:29:49.656387 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.656312    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd3074b6-6505-4bd3-8738-213aa47b12c1/kube-rbac-proxy-thanos/0.log"
Sep 16 19:29:49.693460 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.693425    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd3074b6-6505-4bd3-8738-213aa47b12c1/init-config-reloader/0.log"
Sep 16 19:29:49.742819 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.742790    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-bfm22_705435ab-3aa7-4453-b731-81b340275263/prometheus-operator/0.log"
Sep 16 19:29:49.767984 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.767950    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-bfm22_705435ab-3aa7-4453-b731-81b340275263/kube-rbac-proxy/0.log"
Sep 16 19:29:49.797728 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.797699    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-ppgqc_da54398c-fc8e-43b7-b21c-476d57c695e6/prometheus-operator-admission-webhook/0.log"
Sep 16 19:29:49.837959 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.837931    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75ff4865fc-9mq57_4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154/telemeter-client/0.log"
Sep 16 19:29:49.890043 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.890015    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75ff4865fc-9mq57_4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154/reload/0.log"
Sep 16 19:29:49.948401 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:49.948179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75ff4865fc-9mq57_4793c3bc-0e16-4e1b-8ed9-38cc1cd2b154/kube-rbac-proxy/0.log"
Sep 16 19:29:50.007291 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:50.007264    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b49dff5d-g777h_04aee5ca-0cf7-489a-b9e8-09592e9c12d7/thanos-query/0.log"
Sep 16 19:29:50.060655 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:50.060584    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b49dff5d-g777h_04aee5ca-0cf7-489a-b9e8-09592e9c12d7/kube-rbac-proxy-web/0.log"
Sep 16 19:29:50.101343 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:50.101297    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b49dff5d-g777h_04aee5ca-0cf7-489a-b9e8-09592e9c12d7/kube-rbac-proxy/0.log"
Sep 16 19:29:50.157087 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:50.157044    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b49dff5d-g777h_04aee5ca-0cf7-489a-b9e8-09592e9c12d7/prom-label-proxy/0.log"
Sep 16 19:29:50.214019 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:50.213917    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b49dff5d-g777h_04aee5ca-0cf7-489a-b9e8-09592e9c12d7/kube-rbac-proxy-rules/0.log"
Sep 16 19:29:50.245385 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:50.245355    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b49dff5d-g777h_04aee5ca-0cf7-489a-b9e8-09592e9c12d7/kube-rbac-proxy-metrics/0.log"
Sep 16 19:29:52.060501 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.060459    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"]
Sep 16 19:29:52.061202 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.060978    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" containerName="copy"
Sep 16 19:29:52.061202 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.060997    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" containerName="copy"
Sep 16 19:29:52.061202 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.061021    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" containerName="gather"
Sep 16 19:29:52.061202 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.061030    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" containerName="gather"
Sep 16 19:29:52.061202 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.061132    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" containerName="gather"
Sep 16 19:29:52.061202 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.061145    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="746c0bf3-4784-48e5-aa97-20f2aba0341c" containerName="copy"
Sep 16 19:29:52.064507 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.064485    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.082396 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.082365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"]
Sep 16 19:29:52.119175 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.118708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-lib-modules\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.119175 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.118753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-sys\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.119175 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.118833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-proc\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.119175 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.118886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvspg\" (UniqueName: \"kubernetes.io/projected/62177ca3-0575-4e4d-b7dc-c3284a3bc769-kube-api-access-fvspg\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.119175 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.118918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-podres\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.219497 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.219464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvspg\" (UniqueName: \"kubernetes.io/projected/62177ca3-0575-4e4d-b7dc-c3284a3bc769-kube-api-access-fvspg\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.219679 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.219517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-podres\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.219679 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.219570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-lib-modules\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.219679 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.219595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-sys\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.219679 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.219666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-proc\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.219896 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.219728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-podres\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.219896 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.219735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-lib-modules\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.219896 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.219737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-sys\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.219896 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.219743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62177ca3-0575-4e4d-b7dc-c3284a3bc769-proc\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.230236 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.230209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvspg\" (UniqueName: \"kubernetes.io/projected/62177ca3-0575-4e4d-b7dc-c3284a3bc769-kube-api-access-fvspg\") pod \"perf-node-gather-daemonset-6q6jr\" (UID: \"62177ca3-0575-4e4d-b7dc-c3284a3bc769\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.377242 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.377202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:52.684298 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:52.684206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"]
Sep 16 19:29:52.688469 ip-10-0-140-67 kubenswrapper[2579]: W0916 19:29:52.688431    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod62177ca3_0575_4e4d_b7dc_c3284a3bc769.slice/crio-dac166863485f1355bbf651e5da2387989ab94ac0a1eb10709bf8027e0489b06 WatchSource:0}: Error finding container dac166863485f1355bbf651e5da2387989ab94ac0a1eb10709bf8027e0489b06: Status 404 returned error can't find the container with id dac166863485f1355bbf651e5da2387989ab94ac0a1eb10709bf8027e0489b06
Sep 16 19:29:53.612541 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:53.612494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr" event={"ID":"62177ca3-0575-4e4d-b7dc-c3284a3bc769","Type":"ContainerStarted","Data":"e3ba49e61ba2bf5cd7ecfb914ebcebdbb6b84f696deccce304b5678d6b109995"}
Sep 16 19:29:53.612962 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:53.612597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr" event={"ID":"62177ca3-0575-4e4d-b7dc-c3284a3bc769","Type":"ContainerStarted","Data":"dac166863485f1355bbf651e5da2387989ab94ac0a1eb10709bf8027e0489b06"}
Sep 16 19:29:53.612962 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:53.612628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:53.633497 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:53.633448    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr" podStartSLOduration=1.6334323130000001 podStartE2EDuration="1.633432313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:53.632417303 +0000 UTC m=+2948.297689754" watchObservedRunningTime="2026-09-16 19:29:53.633432313 +0000 UTC m=+2948.298704743"
Sep 16 19:29:54.431791 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:54.431766    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nl6jb_0f297a8b-f692-44f8-8b5c-17a8089acfc2/dns/0.log"
Sep 16 19:29:54.458673 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:54.458645    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nl6jb_0f297a8b-f692-44f8-8b5c-17a8089acfc2/kube-rbac-proxy/0.log"
Sep 16 19:29:54.604380 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:54.604350    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s5dww_9d9a4c61-28b6-4ab1-88c0-4d431578e06f/dns-node-resolver/0.log"
Sep 16 19:29:55.120451 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:55.120419    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l6fts_d65d4d93-e2cc-4ced-b860-e6c1accf08b7/node-ca/0.log"
Sep 16 19:29:56.487642 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:56.487612    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zwxfw_61f31fe1-86ed-4531-a0a8-783318d9dab8/serve-healthcheck-canary/0.log"
Sep 16 19:29:56.999649 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:56.999623    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2h8c_5773ddb2-d163-4f2f-aa26-d128fe807649/kube-rbac-proxy/0.log"
Sep 16 19:29:57.027318 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:57.027292    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2h8c_5773ddb2-d163-4f2f-aa26-d128fe807649/exporter/0.log"
Sep 16 19:29:57.051600 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:57.051573    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2h8c_5773ddb2-d163-4f2f-aa26-d128fe807649/extractor/0.log"
Sep 16 19:29:59.629541 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:59.629508    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-6q6jr"
Sep 16 19:29:59.645900 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:59.645880    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-597x6_63fe53c4-2c29-4698-8cc4-bc7d13210fb1/server/0.log"
Sep 16 19:29:59.747970 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:59.747941    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-4lnvt_e5ea1a68-d91a-4c19-8ead-19bbf99edc8b/manager/0.log"
Sep 16 19:29:59.782874 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:59.782851    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zp7gn_ec391b92-7e84-445b-9344-ba99b3adc52d/s3-init/0.log"
Sep 16 19:29:59.803917 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:29:59.803888    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zp7gn_ec391b92-7e84-445b-9344-ba99b3adc52d/download-hf-model/0.log"
Sep 16 19:30:06.390588 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:06.390557    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw7zd_4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:30:06.446680 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:06.446639    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw7zd_4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b/egress-router-binary-copy/0.log"
Sep 16 19:30:06.509327 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:06.509306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw7zd_4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b/cni-plugins/0.log"
Sep 16 19:30:06.547446 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:06.547426    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw7zd_4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b/bond-cni-plugin/0.log"
Sep 16 19:30:06.580926 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:06.580908    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw7zd_4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b/routeoverride-cni/0.log"
Sep 16 19:30:06.609398 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:06.609375    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw7zd_4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b/whereabouts-cni-bincopy/0.log"
Sep 16 19:30:06.645276 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:06.645228    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw7zd_4125e54e-d784-4ac0-b8b6-c18c5f6d6a5b/whereabouts-cni/0.log"
Sep 16 19:30:06.911785 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:06.911704    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nf4w7_9465c804-604d-44af-be5d-523b881029e8/kube-multus/0.log"
Sep 16 19:30:07.102988 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:07.102896    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-srqqp_8a2b3c86-57c3-4cd2-8142-5be14426c108/network-metrics-daemon/0.log"
Sep 16 19:30:07.128867 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:07.128841    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-srqqp_8a2b3c86-57c3-4cd2-8142-5be14426c108/kube-rbac-proxy/0.log"
Sep 16 19:30:08.178838 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:08.178794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smblz_ac00e65a-d284-4f21-86d0-50dd1b532515/ovn-controller/0.log"
Sep 16 19:30:08.202807 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:08.202780    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smblz_ac00e65a-d284-4f21-86d0-50dd1b532515/ovn-acl-logging/0.log"
Sep 16 19:30:08.235450 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:08.235418    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smblz_ac00e65a-d284-4f21-86d0-50dd1b532515/kube-rbac-proxy-node/0.log"
Sep 16 19:30:08.262936 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:08.262913    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smblz_ac00e65a-d284-4f21-86d0-50dd1b532515/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:30:08.288456 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:08.288434    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smblz_ac00e65a-d284-4f21-86d0-50dd1b532515/northd/0.log"
Sep 16 19:30:08.312121 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:08.312094    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smblz_ac00e65a-d284-4f21-86d0-50dd1b532515/nbdb/0.log"
Sep 16 19:30:08.337605 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:08.337584    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smblz_ac00e65a-d284-4f21-86d0-50dd1b532515/sbdb/0.log"
Sep 16 19:30:08.382126 ip-10-0-140-67 kubenswrapper[2579]: I0916 19:30:08.382099    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smblz_ac00e65a-d284-4f21-86d0-50dd1b532515/ovnkube-controller/0.log"