Sep 18 20:20:34.531836 ip-10-0-138-28 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 20:20:38.516316 ip-10-0-138-28 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:38.516316 ip-10-0-138-28 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 20:20:38.516316 ip-10-0-138-28 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:38.516316 ip-10-0-138-28 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 20:20:38.516316 ip-10-0-138-28 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:38.517269 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.516412    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 20:20:38.520266 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520242    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:38.520266 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520261    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:38.520266 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520266    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:38.520266 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520270    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:38.520266 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520274    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520278    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520283    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520287    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520292    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520296    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520300    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520303    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520307    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520318    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520322    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520326    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520330    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520334    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520339    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520343    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520348    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520354    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520358    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520362    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:38.520582 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520366    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520370    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520374    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520378    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520382    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520387    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520391    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520394    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520398    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520402    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520406    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520410    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520415    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520421    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520428    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520434    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520439    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520444    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520469    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520476    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:38.521405 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520481    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520485    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520490    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520495    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520499    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520502    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520507    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520511    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520515    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520519    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520523    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520527    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520532    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520536    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520540    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520544    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520548    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520552    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520556    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520560    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:38.522338 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520564    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520568    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520572    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520576    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520581    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520584    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520589    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520593    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520606    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520611    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520616    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520620    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520624    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520628    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520633    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520637    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520641    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520645    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520649    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:38.523197 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520653    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520657    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520662    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520666    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520670    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520674    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520678    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520682    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520686    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520690    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520694    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520698    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520702    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520706    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520710    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.520715    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521863    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521874    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521879    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521885    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:38.524027 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521889    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521893    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521902    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521907    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521912    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521921    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521928    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521933    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521938    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521942    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521947    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521952    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521957    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521961    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521966    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521970    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521974    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521978    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521983    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:38.524948 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521987    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521991    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.521996    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522000    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522004    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522008    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522012    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522016    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522021    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522027    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522034    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522039    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522044    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522049    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522053    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522057    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522062    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522067    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522071    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:38.525563 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522076    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522080    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522084    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522088    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522092    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522096    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522101    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522105    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522109    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522114    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522118    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522122    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522129    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522134    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522138    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522142    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522147    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522151    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522155    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:38.526108 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522160    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522164    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522168    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522173    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522177    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522181    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522185    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522190    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522194    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522199    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522203    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522208    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522212    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522218    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522222    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522226    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522231    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522236    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522240    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522244    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:38.526611 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522248    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522252    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522256    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522261    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522265    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522270    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522274    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522278    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522282    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522287    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522291    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522295    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522299    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522304    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522309    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522313    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522317    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522321    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522929    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522938    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:38.527118 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522942    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522947    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522951    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522956    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522961    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522965    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522969    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522974    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522978    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522982    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522986    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522990    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522994    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.522999    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523003    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523007    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523011    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523015    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523019    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523023    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:38.527786 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523028    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523032    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523036    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523041    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523045    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523050    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523054    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523058    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523062    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523067    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523071    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523075    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523079    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523083    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523088    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523095    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523102    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523107    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523112    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:38.528332 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523117    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523121    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523126    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523132    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523137    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523141    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523145    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523149    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523154    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523160    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523165    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523170    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523175    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523180    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523184    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523188    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523193    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523197    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523201    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:38.528882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523205    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523209    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523213    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523218    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523225    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523232    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523236    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523240    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523244    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523249    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523253    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523257    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523261    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523265    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523269    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523273    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523278    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523282    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523286    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523291    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:38.529363 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523297    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523301    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523305    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523309    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523313    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523317    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523321    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523325    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523332    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523336    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523340    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523344    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523348    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523352    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523357    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523361    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523366    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523370    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.523374    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523496    2557 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523547    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 20:20:38.529877 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523557    2557 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523564    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523571    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523577    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523584    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523590    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523596    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523600    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523606    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523611    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523616    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523621    2557 flags.go:64] FLAG: --cgroup-root=""
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523626    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523631    2557 flags.go:64] FLAG: --client-ca-file=""
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523635    2557 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523640    2557 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523646    2557 flags.go:64] FLAG: --cluster-domain=""
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523651    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523655    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523663    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523669    2557 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523675    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523680    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 20:20:38.530379 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523684    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523690    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523695    2557 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523699    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523706    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523711    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523716    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523722    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523727    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523732    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523737    2557 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523741    2557 flags.go:64] FLAG: --enable-server="true"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523746    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523756    2557 flags.go:64] FLAG: --event-burst="100"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523761    2557 flags.go:64] FLAG: --event-qps="50"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523765    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523770    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523775    2557 flags.go:64] FLAG: --eviction-hard=""
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523781    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523786    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523790    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523795    2557 flags.go:64] FLAG: --eviction-soft=""
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523800    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523805    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523809    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 20:20:38.530935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523814    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523818    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523823    2557 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523831    2557 flags.go:64] FLAG: --feature-gates=""
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523837    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523842    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523848    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523853    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523857    2557 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523862    2557 flags.go:64] FLAG: --help="false"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523867    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523873    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523878    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523883    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523889    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523895    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523899    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523904    2557 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523908    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523913    2557 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523918    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523924    2557 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523928    2557 flags.go:64] FLAG: --kube-reserved=""
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523933    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 20:20:38.531698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523938    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523942    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523947    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523951    2557 flags.go:64] FLAG: --lock-file=""
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523976    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523982    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523988    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.523997    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524003    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524008    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524012    2557 flags.go:64] FLAG: --logging-format="text"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524020    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524026    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524030    2557 flags.go:64] FLAG: --manifest-url=""
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524034    2557 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524041    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524046    2557 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524053    2557 flags.go:64] FLAG: --max-pods="110"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524057    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524064    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524069    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524074    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524078    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524083    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524088    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 20:20:38.532578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524099    2557 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524104    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524108    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524113    2557 flags.go:64] FLAG: --pod-cidr=""
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524118    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524128    2557 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524132    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524137    2557 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524142    2557 flags.go:64] FLAG: --port="10250"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524147    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524151    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b4d4182df42ca366"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524156    2557 flags.go:64] FLAG: --qos-reserved=""
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524161    2557 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524166    2557 flags.go:64] FLAG: --register-node="true"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524170    2557 flags.go:64] FLAG: --register-with-taints=""
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524176    2557 flags.go:64] FLAG: --registry-burst="10"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524181    2557 flags.go:64] FLAG: --registry-qps="5"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524186    2557 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524190    2557 flags.go:64] FLAG: --reserved-memory=""
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524198    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524203    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524208    2557 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524213    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524217    2557 flags.go:64] FLAG: --runonce="false"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524222    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 20:20:38.533319 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524227    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524234    2557 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524239    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524243    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524248    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524253    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524258    2557 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524263    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524267    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524272    2557 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524276    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524281    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524285    2557 flags.go:64] FLAG: --system-cgroups=""
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524291    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524300    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524305    2557 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524309    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524315    2557 flags.go:64] FLAG: --tls-min-version=""
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524319    2557 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524324    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524329    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524333    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524338    2557 flags.go:64] FLAG: --v="2"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524345    2557 flags.go:64] FLAG: --version="false"
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524352    2557 flags.go:64] FLAG: --vmodule=""
Sep 18 20:20:38.533985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524359    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.524366    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524562    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524570    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524574    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524579    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524584    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524589    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524596    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524601    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524605    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524610    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524614    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524618    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524623    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524627    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524631    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524635    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524639    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524644    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:38.534608 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524648    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524653    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524660    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524666    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524670    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524675    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524679    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524683    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524687    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524691    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524695    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524700    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524704    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524710    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524715    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524719    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524723    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524727    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524732    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524736    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:38.535090 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524742    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524746    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524753    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524758    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524763    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524768    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524772    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524777    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524781    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524786    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524791    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524796    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524800    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524804    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524809    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524813    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524817    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524821    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524825    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:38.535693 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524829    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524834    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524838    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524845    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524850    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524855    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524860    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524865    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524870    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524874    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524878    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524883    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524887    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524893    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524897    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524901    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524905    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524909    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524914    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524919    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:38.536163 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524923    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524927    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524932    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524936    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524940    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524944    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524948    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524952    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524956    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524960    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524964    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524968    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524972    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524977    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524982    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524986    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524991    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.524995    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.525001    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.525005    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:38.536663 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.525009    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.525013    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.525654    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.534875    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.534890    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.534915    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.534921    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.534989    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.534993    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.534997    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535000    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535003    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535007    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535009    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535012    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:38.537148 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535015    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535018    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535020    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535023    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535026    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535031    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535035    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535039    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535042    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535046    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535049    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535053    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535056    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535059    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535062    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535065    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535068    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535071    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535073    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:38.537554 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535076    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535079    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535082    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535085    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535088    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535091    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535093    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535096    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535099    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535102    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535105    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535108    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535111    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535114    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535116    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535119    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535121    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535124    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535127    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535130    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:38.538023 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535132    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535135    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535137    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535140    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535142    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535144    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535147    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535150    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535153    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535155    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535158    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535161    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535163    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535166    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535169    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535172    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535175    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535177    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535180    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535182    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:38.538527 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535185    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535188    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535190    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535193    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535195    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535198    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535202    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535205    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535208    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535211    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535214    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535217    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535220    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535223    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535226    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535229    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535231    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535234    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535237    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:38.539045 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535239    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535242    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535244    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535248    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535250    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535253    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535256    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535259    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535261    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535264    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535267    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535269    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535272    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.535277    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535409    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:38.539519 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535414    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535418    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535421    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535423    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535426    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535429    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535432    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535434    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535437    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535439    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535442    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535445    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535448    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535465    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535470    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535472    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535475    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535478    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535481    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535483    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:38.539883 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535486    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535489    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535492    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535494    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535498    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535501    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535503    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535506    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535509    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535511    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535514    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535517    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535520    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535522    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535526    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535530    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535533    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535536    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535539    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:38.540373 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535542    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535545    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535547    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535550    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535552    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535555    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535558    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535561    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535563    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535566    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535568    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535571    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535573    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535576    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535579    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535582    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535584    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535588    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535592    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535595    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:38.540922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535597    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535600    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535602    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535605    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535609    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535612    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535615    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535618    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535621    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535623    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535626    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535628    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535631    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535633    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535636    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535639    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535641    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535644    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535648    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:38.541404 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535650    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535653    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535655    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535658    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535660    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535665    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535668    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535670    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535673    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535675    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535678    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535681    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535684    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535686    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535689    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535691    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535694    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535697    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535700    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:38.541882 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:38.535702    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:38.542335 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.535707    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:38.542335 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.535806    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 20:20:38.542335 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.537807    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 20:20:38.542335 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.538758    2557 server.go:1019] "Starting client certificate rotation"
Sep 18 20:20:38.542335 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.538860    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 20:20:38.542335 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.538900    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 20:20:38.542335 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.541272    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 20:20:38.542537 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.542373    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 20:20:38.546523 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.546502    2557 log.go:25] "Validated CRI v1 runtime API"
Sep 18 20:20:38.548427 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.548412    2557 log.go:25] "Validated CRI v1 image API"
Sep 18 20:20:38.549231 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.549212    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 20:20:38.551466 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.551431    2557 fs.go:135] Filesystem UUIDs: map[0293f945-ad18-4c41-8e3a-d6fdb8e3abd2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b35056de-9b20-4fea-8b9b-f54bc86a410a:/dev/nvme0n1p3]
Sep 18 20:20:38.551551 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.551468    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 20:20:38.556019 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.555897    2557 manager.go:216] Machine: {Timestamp:2026-09-18 20:20:38.555322928 +0000 UTC m=+2.048675499 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101251 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a32d2b33675eb165b271ccfc283b SystemUUID:ec28a32d-2b33-675e-b165-b271ccfc283b BootID:1e7c2c2f-cffc-451f-95e8-291dccfb9daa Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:f4:00:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:f4:00:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:ec:85:3a:71:40 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 20:20:38.556019 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.556018    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 20:20:38.556121 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.556091    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 20:20:38.558231 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.558211    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 20:20:38.558380 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.558235    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 20:20:38.560155 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.560142    2557 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 20:20:38.560195 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.560160    2557 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 20:20:38.560195 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.560175    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 20:20:38.560250 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.560194    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 20:20:38.561414 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.561403    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 20:20:38.565018 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.565006    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 20:20:38.565771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.565749    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 20:20:38.565861 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.565824    2557 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 20:20:38.565921 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.565865    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 20:20:38.565921 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.565883    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 20:20:38.565921 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.565897    2557 kubelet.go:398] "Adding apiserver pod source"
Sep 18 20:20:38.565921 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.565909    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 20:20:38.566472 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.566435    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 20:20:38.566549 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.566497    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 20:20:38.569969 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.569950    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 20:20:38.570117 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.570099    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 20:20:38.571999 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.571985    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 20:20:38.572057 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.572015    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 20:20:38.572189 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.572177    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 20:20:38.572230 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.572193    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 20:20:38.572230 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.572199    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 20:20:38.572230 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.572205    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 20:20:38.572230 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.572213    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 20:20:38.572230 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.572218    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 20:20:38.573330 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.573320    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 20:20:38.573330 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.573330    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 20:20:38.573400 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.573336    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 20:20:38.573400 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.573342    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 20:20:38.573400 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.573350    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 20:20:38.573400 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.573358    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 20:20:38.573400 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.573381    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 20:20:38.573400 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.573386    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 20:20:38.576992 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.576978    2557 server.go:1268] "Started kubelet"
Sep 18 20:20:38.577097 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.577070    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 20:20:38.577266 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.577224    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 20:20:38.577333 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.577286    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 20:20:38.577965 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.577792    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 20:20:38.577965 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.577924    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 20:20:38.577959 ip-10-0-138-28 systemd[1]: Started Kubernetes Kubelet.
Sep 18 20:20:38.578559 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.577984    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 20:20:38.578559 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.578023    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 20:20:38.578559 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.578489    2557 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 20:20:38.582191 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.582173    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 20:20:38.585354 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.585339    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 20:20:38.585402 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.585363    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 20:20:38.585815 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.585528    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:38.585912 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.585896    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 20:20:38.585967 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.585917    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 20:20:38.586017 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.585887    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 20:20:38.586057 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.586041    2557 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 20:20:38.586057 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.586048    2557 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 20:20:38.586189 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.586116    2557 factory.go:55] Registering systemd factory
Sep 18 20:20:38.586189 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.586134    2557 factory.go:223] Registration of the systemd container factory successfully
Sep 18 20:20:38.586339 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.586322    2557 factory.go:153] Registering CRI-O factory
Sep 18 20:20:38.586339 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.586341    2557 factory.go:223] Registration of the crio container factory successfully
Sep 18 20:20:38.586512 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.586408    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 20:20:38.586512 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.586426    2557 factory.go:103] Registering Raw factory
Sep 18 20:20:38.586512 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.586442    2557 manager.go:1196] Started watching for new ooms in manager
Sep 18 20:20:38.586977 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.586964    2557 manager.go:349] Starting recovery of all containers
Sep 18 20:20:38.587917 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.587892    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 20:20:38.587999 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.587981    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 20:20:38.613360 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.613336    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sfm8p"
Sep 18 20:20:38.613445 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.613359    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sfm8p"
Sep 18 20:20:38.620585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.620443    2557 manager.go:354] Recovery completed
Sep 18 20:20:38.624625 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.624611    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:38.627129 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.627113    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:38.627223 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.627141    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:38.627223 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.627155    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:38.627733 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.627717    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 20:20:38.627733 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.627730    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 20:20:38.627828 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.627744    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 20:20:38.627828 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.627765    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:38.628443 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.628430    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:38.628505 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.628467    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:38.628505 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.628477    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:38.629770 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.629757    2557 policy_none.go:49] "None policy: Start"
Sep 18 20:20:38.629820 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.629773    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 20:20:38.629820 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.629782    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 20:20:38.631711 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.631701    2557 policy_none.go:47] "Start"
Sep 18 20:20:38.682072 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.682057    2557 manager.go:337] "Starting Device Plugin manager"
Sep 18 20:20:38.685322 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.682090    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 20:20:38.685322 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.682102    2557 server.go:87] "Starting device plugin registration server"
Sep 18 20:20:38.685322 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.682321    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 20:20:38.685322 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.682338    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 20:20:38.685322 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.682430    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 20:20:38.685322 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.682529    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 20:20:38.685322 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.682537    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 20:20:38.685322 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.682968    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 20:20:38.685322 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.683011    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:38.715294 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.715264    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 20:20:38.716420 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.716406    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 20:20:38.716509 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.716449    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 20:20:38.716509 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.716488    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 20:20:38.716597 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.716531    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 20:20:38.718478 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.718445    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:38.783500 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.783439    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:38.784531 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.784510    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:38.784616 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.784543    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:38.784616 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.784554    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:38.784616 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.784577    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.792926 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.792912    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.792970 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.792933    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-28.ec2.internal\": node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:38.811147 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.811130    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:38.817724 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.817700    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal"]
Sep 18 20:20:38.817772 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.817764    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:38.818603 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.818586    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:38.818655 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.818619    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:38.818655 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.818629    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:38.820583 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.820572    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:38.820710 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.820695    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.820747 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.820726    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:38.821301 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.821289    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:38.821358 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.821312    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:38.821358 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.821321    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:38.821566 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.821550    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:38.821634 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.821577    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:38.821634 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.821601    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:38.823690 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.823676    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.823772 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.823699    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:38.824373 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.824357    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:38.824449 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.824383    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:38.824449 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.824396    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:38.844984 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.844966    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-28.ec2.internal\" not found" node="ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.848919 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.848906    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-28.ec2.internal\" not found" node="ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.887252 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.887234    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee72dde8fdd947ba26738b5722deca91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal\" (UID: \"ee72dde8fdd947ba26738b5722deca91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.887304 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.887258    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee72dde8fdd947ba26738b5722deca91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal\" (UID: \"ee72dde8fdd947ba26738b5722deca91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.887304 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.887273    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7d826f4c7a124136216df58c6a6c0d1b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-28.ec2.internal\" (UID: \"7d826f4c7a124136216df58c6a6c0d1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.911622 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:38.911606    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:38.987811 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.987788    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee72dde8fdd947ba26738b5722deca91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal\" (UID: \"ee72dde8fdd947ba26738b5722deca91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.987903 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.987823    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee72dde8fdd947ba26738b5722deca91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal\" (UID: \"ee72dde8fdd947ba26738b5722deca91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.987903 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.987840    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee72dde8fdd947ba26738b5722deca91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal\" (UID: \"ee72dde8fdd947ba26738b5722deca91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.987903 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.987848    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7d826f4c7a124136216df58c6a6c0d1b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-28.ec2.internal\" (UID: \"7d826f4c7a124136216df58c6a6c0d1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.987903 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.987877    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee72dde8fdd947ba26738b5722deca91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal\" (UID: \"ee72dde8fdd947ba26738b5722deca91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:38.987903 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:38.987877    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7d826f4c7a124136216df58c6a6c0d1b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-28.ec2.internal\" (UID: \"7d826f4c7a124136216df58c6a6c0d1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:39.011917 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:39.011896    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:39.112652 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:39.112606    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:39.147768 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.147740    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:39.153124 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.152848    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:39.213309 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:39.213284    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:39.313772 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:39.313745    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:39.414298 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:39.414281    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:39.441905 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.441885    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:39.515133 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:39.515111    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:39.538271 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.538251    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 20:20:39.538773 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.538367    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 20:20:39.538773 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.538400    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 20:20:39.616061 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:39.616040    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-28.ec2.internal\" not found"
Sep 18 20:20:39.619218 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.619188    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 20:15:38 +0000 UTC" deadline="2028-03-19 08:55:42.738334369 +0000 UTC"
Sep 18 20:20:39.619315 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.619221    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13140h35m3.119118428s"
Sep 18 20:20:39.695490 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.695432    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:39.776371 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:39.776341    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d826f4c7a124136216df58c6a6c0d1b.slice/crio-e79d5e1ef329d3eebbc7212cb7c007823950dff68b5b411ba46cae98f3388c1e WatchSource:0}: Error finding container e79d5e1ef329d3eebbc7212cb7c007823950dff68b5b411ba46cae98f3388c1e: Status 404 returned error can't find the container with id e79d5e1ef329d3eebbc7212cb7c007823950dff68b5b411ba46cae98f3388c1e
Sep 18 20:20:39.776798 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:39.776779    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee72dde8fdd947ba26738b5722deca91.slice/crio-f28fa91cc04ef940b3dda8da6aaee4f81045e3907dab77d5c7df911f61ce4362 WatchSource:0}: Error finding container f28fa91cc04ef940b3dda8da6aaee4f81045e3907dab77d5c7df911f61ce4362: Status 404 returned error can't find the container with id f28fa91cc04ef940b3dda8da6aaee4f81045e3907dab77d5c7df911f61ce4362
Sep 18 20:20:39.781407 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.781392    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:20:39.785693 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.785677    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:39.795812 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.795796    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 20:20:39.798104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.798088    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal"
Sep 18 20:20:39.805841 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:39.805827    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 20:20:40.069879 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.069802    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:40.567890 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.567821    2557 apiserver.go:52] "Watching apiserver"
Sep 18 20:20:40.574562 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.574483    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 20:20:40.574887 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.574859    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-25n4b","kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal","openshift-cluster-node-tuning-operator/tuned-w2rft","openshift-multus/multus-additional-cni-plugins-hl4v9","openshift-ovn-kubernetes/ovnkube-node-m5vk2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k","openshift-dns/node-resolver-v2ks2","openshift-image-registry/node-ca-jnctc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal","openshift-multus/multus-b6xk6","openshift-multus/network-metrics-daemon-7kdrq","openshift-network-diagnostics/network-check-target-bm5dj","openshift-network-operator/iptables-alerter-zlggb","kube-system/global-pull-secret-syncer-l6x2p"]
Sep 18 20:20:40.577540 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.577510    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.580097 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.579580    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.580097 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.579924    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hm8sg\""
Sep 18 20:20:40.580097 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.579948    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 20:20:40.580097 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.579928    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 20:20:40.582044 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.582003    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7wmfv\""
Sep 18 20:20:40.582157 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.582054    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 20:20:40.582157 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.582054    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 20:20:40.582296 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.582161    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 20:20:40.582357 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.582334    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.584142 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.584124    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.584664 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.584626    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8q7w8\""
Sep 18 20:20:40.584664 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.584642    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 20:20:40.584817 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.584640    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 20:20:40.586795 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.586143    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 20:20:40.586795 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.586349    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 20:20:40.586795 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.586355    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 20:20:40.586795 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.586630    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gdd7j\""
Sep 18 20:20:40.586795 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.586637    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 20:20:40.586795 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.586794    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 20:20:40.587148 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.586858    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.587148 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.586880    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 20:20:40.590669 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.589520    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.590669 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.590384    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 20:20:40.590812 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.590678    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 20:20:40.590812 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.590727    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 20:20:40.590812 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.590802    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 20:20:40.590969 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.590805    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 20:20:40.590969 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.590946    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 20:20:40.591352 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.591144    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gcdtx\""
Sep 18 20:20:40.591470 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.591421    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 20:20:40.593136 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.591953    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 20:20:40.593136 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.591992    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 20:20:40.593136 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.592293    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:40.593136 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.592511    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8hw2t\""
Sep 18 20:20:40.593136 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.592473    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595268    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6g9\" (UniqueName: \"kubernetes.io/projected/fcceb3e3-7696-404d-a730-edc53df512cc-kube-api-access-hb6g9\") pod \"node-resolver-v2ks2\" (UID: \"fcceb3e3-7696-404d-a730-edc53df512cc\") " pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595568    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-modprobe-d\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595590    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-run\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-host\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595644    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b2d5d99-7944-4e3c-95f7-a849c81acda1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595666    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-systemd-units\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595687    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86c0c73b-2139-4f34-9bc6-a972fb211a9f-host-slash\") pod \"iptables-alerter-zlggb\" (UID: \"86c0c73b-2139-4f34-9bc6-a972fb211a9f\") " pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595707    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79598812-762d-49dc-94e2-91f9525c69a9-tmp\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595729    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b2d5d99-7944-4e3c-95f7-a849c81acda1-cnibin\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595750    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-cni-netd\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595772    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595801    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c03a0417-3da2-4804-b145-74d2f4954f2c-env-overrides\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595822    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c03a0417-3da2-4804-b145-74d2f4954f2c-ovn-node-metrics-cert\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595843    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c03a0417-3da2-4804-b145-74d2f4954f2c-ovnkube-script-lib\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595866    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b2d5d99-7944-4e3c-95f7-a849c81acda1-cni-binary-copy\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.597662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595899    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b2d5d99-7944-4e3c-95f7-a849c81acda1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8kwx\" (UniqueName: \"kubernetes.io/projected/9b2d5d99-7944-4e3c-95f7-a849c81acda1-kube-api-access-g8kwx\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595943    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-kubelet\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-etc-openvswitch\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.595991    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-log-socket\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596013    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcceb3e3-7696-404d-a730-edc53df512cc-hosts-file\") pod \"node-resolver-v2ks2\" (UID: \"fcceb3e3-7696-404d-a730-edc53df512cc\") " pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596056    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86c0c73b-2139-4f34-9bc6-a972fb211a9f-iptables-alerter-script\") pod \"iptables-alerter-zlggb\" (UID: \"86c0c73b-2139-4f34-9bc6-a972fb211a9f\") " pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-run-systemd\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596099    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-run-ovn\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596120    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c03a0417-3da2-4804-b145-74d2f4954f2c-ovnkube-config\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596142    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b2d5d99-7944-4e3c-95f7-a849c81acda1-system-cni-dir\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-sysctl-d\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596188    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-sysctl-conf\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596209    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvtn\" (UniqueName: \"kubernetes.io/projected/79598812-762d-49dc-94e2-91f9525c69a9-kube-api-access-9lvtn\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88q95\" (UniqueName: \"kubernetes.io/projected/c03a0417-3da2-4804-b145-74d2f4954f2c-kube-api-access-88q95\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.598741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596257    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzrwd\" (UniqueName: \"kubernetes.io/projected/86c0c73b-2139-4f34-9bc6-a972fb211a9f-kube-api-access-pzrwd\") pod \"iptables-alerter-zlggb\" (UID: \"86c0c73b-2139-4f34-9bc6-a972fb211a9f\") " pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596278    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-sys\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596298    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-lib-modules\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596322    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-var-lib-kubelet\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596342    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79598812-762d-49dc-94e2-91f9525c69a9-etc-tuned\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596364    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b2d5d99-7944-4e3c-95f7-a849c81acda1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-node-log\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596406    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-cni-bin\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596427    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcceb3e3-7696-404d-a730-edc53df512cc-tmp-dir\") pod \"node-resolver-v2ks2\" (UID: \"fcceb3e3-7696-404d-a730-edc53df512cc\") " pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596463    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-sysconfig\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596489    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-kubernetes\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596510    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-systemd\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596531    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-slash\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596551    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-run-netns\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596573    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-var-lib-openvswitch\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.596594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-run-openvswitch\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.598357    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.599439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.599238    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j9ndn\""
Sep 18 20:20:40.600233 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.599643    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 20:20:40.600233 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.599817    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 20:20:40.600233 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.599876    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.600565 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.600365    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 20:20:40.601895 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.601472    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 20:20:40.601895 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.601634    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vk2x9\""
Sep 18 20:20:40.601895 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.601664    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 20:20:40.602241 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.602223    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5gvkm\""
Sep 18 20:20:40.603048 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.602911    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 20:20:40.603153 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.603067    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 20:20:40.603153 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.603083    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:40.603271 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.603151    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:20:40.605000 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.604888    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:40.605000 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.604952    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:20:40.620236 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.620218    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cf2rb"
Sep 18 20:20:40.627052 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.627035    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cf2rb"
Sep 18 20:20:40.687040 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.687016    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 20:20:40.696937 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.696919    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-log-socket\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.697037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.696944    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6g9\" (UniqueName: \"kubernetes.io/projected/fcceb3e3-7696-404d-a730-edc53df512cc-kube-api-access-hb6g9\") pod \"node-resolver-v2ks2\" (UID: \"fcceb3e3-7696-404d-a730-edc53df512cc\") " pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.697037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.696960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-modprobe-d\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.697037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.696975    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-run\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.697037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.696989    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-host\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.697037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697009    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b2d5d99-7944-4e3c-95f7-a849c81acda1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.697037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-cni-netd\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697053    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-log-socket\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c03a0417-3da2-4804-b145-74d2f4954f2c-env-overrides\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697095    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-run\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697105    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c03a0417-3da2-4804-b145-74d2f4954f2c-ovnkube-script-lib\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697120    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-modprobe-d\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697132    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-host\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697180    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-cni-netd\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697191    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1fa00627-119a-42c6-aff3-568d9234c049-konnectivity-ca\") pod \"konnectivity-agent-25n4b\" (UID: \"1fa00627-119a-42c6-aff3-568d9234c049\") " pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697227    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86c0c73b-2139-4f34-9bc6-a972fb211a9f-host-slash\") pod \"iptables-alerter-zlggb\" (UID: \"86c0c73b-2139-4f34-9bc6-a972fb211a9f\") " pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697283    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86c0c73b-2139-4f34-9bc6-a972fb211a9f-host-slash\") pod \"iptables-alerter-zlggb\" (UID: \"86c0c73b-2139-4f34-9bc6-a972fb211a9f\") " pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697272    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b2d5d99-7944-4e3c-95f7-a849c81acda1-cnibin\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697325    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b2d5d99-7944-4e3c-95f7-a849c81acda1-cnibin\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697324    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697365    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.697374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697376    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-device-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697438    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-run-multus-certs\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697485    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b2d5d99-7944-4e3c-95f7-a849c81acda1-cni-binary-copy\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697530    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b2d5d99-7944-4e3c-95f7-a849c81acda1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8kwx\" (UniqueName: \"kubernetes.io/projected/9b2d5d99-7944-4e3c-95f7-a849c81acda1-kube-api-access-g8kwx\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697647    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c03a0417-3da2-4804-b145-74d2f4954f2c-env-overrides\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697726    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c03a0417-3da2-4804-b145-74d2f4954f2c-ovnkube-script-lib\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697771    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b2d5d99-7944-4e3c-95f7-a849c81acda1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-kubelet\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697851    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-kubelet\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697855    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62a9d73d-1877-48b9-94e2-5a76311b7453-kubelet-config\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62a9d73d-1877-48b9-94e2-5a76311b7453-dbus\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-run-systemd\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697947    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-run-ovn\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697970    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-cnibin\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697970    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-run-systemd\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.698213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.697991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-sysctl-d\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698018    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b2d5d99-7944-4e3c-95f7-a849c81acda1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698122    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-run-ovn\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698132    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-sysctl-d\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-lib-modules\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79598812-762d-49dc-94e2-91f9525c69a9-etc-tuned\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698186    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4fbb\" (UniqueName: \"kubernetes.io/projected/055c8e5b-8db0-441b-a338-08f6572fe0ae-kube-api-access-z4fbb\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698211    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37093cb5-2840-4b9c-97b6-f66bbd717c9c-cni-binary-copy\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-lib-modules\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698233    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-run-netns\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698263    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-etc-kubernetes\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698286    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc34fc3b-3dee-4485-87ba-988209781305-host\") pod \"node-ca-jnctc\" (UID: \"cc34fc3b-3dee-4485-87ba-988209781305\") " pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl8mt\" (UniqueName: \"kubernetes.io/projected/4e506cbe-bfeb-4614-b6c7-91b349785053-kube-api-access-bl8mt\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698336    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw26h\" (UniqueName: \"kubernetes.io/projected/37093cb5-2840-4b9c-97b6-f66bbd717c9c-kube-api-access-jw26h\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698392    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-sysconfig\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-slash\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698446    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b2d5d99-7944-4e3c-95f7-a849c81acda1-cni-binary-copy\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698490    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-run-netns\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698502    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-sysconfig\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698523    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-slash\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698531    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-run-netns\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698532    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c03a0417-3da2-4804-b145-74d2f4954f2c-ovn-node-metrics-cert\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698546    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698572    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-registration-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698601    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc34fc3b-3dee-4485-87ba-988209781305-serviceca\") pod \"node-ca-jnctc\" (UID: \"cc34fc3b-3dee-4485-87ba-988209781305\") " pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698632    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-systemd-units\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698667    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88q95\" (UniqueName: \"kubernetes.io/projected/c03a0417-3da2-4804-b145-74d2f4954f2c-kube-api-access-88q95\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698681    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-systemd-units\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698690    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79598812-762d-49dc-94e2-91f9525c69a9-tmp\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698760    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-var-lib-cni-bin\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698783    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c03a0417-3da2-4804-b145-74d2f4954f2c-ovnkube-config\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698807    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-etc-openvswitch\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.699914 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698838    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698860    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-var-lib-kubelet\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698899    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcceb3e3-7696-404d-a730-edc53df512cc-hosts-file\") pod \"node-resolver-v2ks2\" (UID: \"fcceb3e3-7696-404d-a730-edc53df512cc\") " pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698896    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-etc-openvswitch\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698947    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86c0c73b-2139-4f34-9bc6-a972fb211a9f-iptables-alerter-script\") pod \"iptables-alerter-zlggb\" (UID: \"86c0c73b-2139-4f34-9bc6-a972fb211a9f\") " pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.698981    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1fa00627-119a-42c6-aff3-568d9234c049-agent-certs\") pod \"konnectivity-agent-25n4b\" (UID: \"1fa00627-119a-42c6-aff3-568d9234c049\") " pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699008    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-socket-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699033    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-sys-fs\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-socket-dir-parent\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699080    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b2d5d99-7944-4e3c-95f7-a849c81acda1-system-cni-dir\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-sysctl-conf\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699119    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvtn\" (UniqueName: \"kubernetes.io/projected/79598812-762d-49dc-94e2-91f9525c69a9-kube-api-access-9lvtn\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699152    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699172    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-system-cni-dir\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699193    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzrwd\" (UniqueName: \"kubernetes.io/projected/86c0c73b-2139-4f34-9bc6-a972fb211a9f-kube-api-access-pzrwd\") pod \"iptables-alerter-zlggb\" (UID: \"86c0c73b-2139-4f34-9bc6-a972fb211a9f\") " pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699212    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-sys\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-var-lib-kubelet\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.700654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699251    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b2d5d99-7944-4e3c-95f7-a849c81acda1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699269    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-node-log\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699288    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-run-k8s-cni-cncf-io\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-var-lib-cni-multus\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699328    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmqv\" (UniqueName: \"kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv\") pod \"network-check-target-bm5dj\" (UID: \"edb74921-50a4-402a-b2cc-138778d94d25\") " pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-cni-dir\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86c0c73b-2139-4f34-9bc6-a972fb211a9f-iptables-alerter-script\") pod \"iptables-alerter-zlggb\" (UID: \"86c0c73b-2139-4f34-9bc6-a972fb211a9f\") " pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699375    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-hostroot\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699399    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-conf-dir\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699417    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcceb3e3-7696-404d-a730-edc53df512cc-hosts-file\") pod \"node-resolver-v2ks2\" (UID: \"fcceb3e3-7696-404d-a730-edc53df512cc\") " pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699327    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c03a0417-3da2-4804-b145-74d2f4954f2c-ovnkube-config\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699472    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b2d5d99-7944-4e3c-95f7-a849c81acda1-system-cni-dir\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699486    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-sys\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699598    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-var-lib-kubelet\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699625    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-daemon-config\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699642    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-node-log\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.701326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699648    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-sysctl-conf\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699678    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-cni-bin\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699704    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f92dc\" (UniqueName: \"kubernetes.io/projected/cc34fc3b-3dee-4485-87ba-988209781305-kube-api-access-f92dc\") pod \"node-ca-jnctc\" (UID: \"cc34fc3b-3dee-4485-87ba-988209781305\") " pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699725    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b2d5d99-7944-4e3c-95f7-a849c81acda1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699734    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-host-cni-bin\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699761    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcceb3e3-7696-404d-a730-edc53df512cc-tmp-dir\") pod \"node-resolver-v2ks2\" (UID: \"fcceb3e3-7696-404d-a730-edc53df512cc\") " pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699796    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-kubernetes\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699881    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-systemd\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699910    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-kubernetes\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699932    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79598812-762d-49dc-94e2-91f9525c69a9-etc-systemd\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699954    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-var-lib-openvswitch\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.699993    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-var-lib-openvswitch\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.700023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-run-openvswitch\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.700038    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcceb3e3-7696-404d-a730-edc53df512cc-tmp-dir\") pod \"node-resolver-v2ks2\" (UID: \"fcceb3e3-7696-404d-a730-edc53df512cc\") " pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.700051    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c03a0417-3da2-4804-b145-74d2f4954f2c-run-openvswitch\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.701913    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79598812-762d-49dc-94e2-91f9525c69a9-tmp\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.701937    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79598812-762d-49dc-94e2-91f9525c69a9-etc-tuned\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.702104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.702045    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c03a0417-3da2-4804-b145-74d2f4954f2c-ovn-node-metrics-cert\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.714287 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.714257    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvtn\" (UniqueName: \"kubernetes.io/projected/79598812-762d-49dc-94e2-91f9525c69a9-kube-api-access-9lvtn\") pod \"tuned-w2rft\" (UID: \"79598812-762d-49dc-94e2-91f9525c69a9\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.714556 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.714537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzrwd\" (UniqueName: \"kubernetes.io/projected/86c0c73b-2139-4f34-9bc6-a972fb211a9f-kube-api-access-pzrwd\") pod \"iptables-alerter-zlggb\" (UID: \"86c0c73b-2139-4f34-9bc6-a972fb211a9f\") " pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.714676 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.714625    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8kwx\" (UniqueName: \"kubernetes.io/projected/9b2d5d99-7944-4e3c-95f7-a849c81acda1-kube-api-access-g8kwx\") pod \"multus-additional-cni-plugins-hl4v9\" (UID: \"9b2d5d99-7944-4e3c-95f7-a849c81acda1\") " pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.714771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.714695    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6g9\" (UniqueName: \"kubernetes.io/projected/fcceb3e3-7696-404d-a730-edc53df512cc-kube-api-access-hb6g9\") pod \"node-resolver-v2ks2\" (UID: \"fcceb3e3-7696-404d-a730-edc53df512cc\") " pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.714887 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.714868    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88q95\" (UniqueName: \"kubernetes.io/projected/c03a0417-3da2-4804-b145-74d2f4954f2c-kube-api-access-88q95\") pod \"ovnkube-node-m5vk2\" (UID: \"c03a0417-3da2-4804-b145-74d2f4954f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.721633 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.721587    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal" event={"ID":"ee72dde8fdd947ba26738b5722deca91","Type":"ContainerStarted","Data":"f28fa91cc04ef940b3dda8da6aaee4f81045e3907dab77d5c7df911f61ce4362"}
Sep 18 20:20:40.723096 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.723073    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal" event={"ID":"7d826f4c7a124136216df58c6a6c0d1b","Type":"ContainerStarted","Data":"e79d5e1ef329d3eebbc7212cb7c007823950dff68b5b411ba46cae98f3388c1e"}
Sep 18 20:20:40.785612 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.785588    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:40.800325 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800302    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc34fc3b-3dee-4485-87ba-988209781305-serviceca\") pod \"node-ca-jnctc\" (UID: \"cc34fc3b-3dee-4485-87ba-988209781305\") " pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.800442 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800333    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.800442 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-var-lib-cni-bin\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800442 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-var-lib-kubelet\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800594 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800442    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.800594 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800444    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-var-lib-cni-bin\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800594 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-var-lib-kubelet\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800594 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800503    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1fa00627-119a-42c6-aff3-568d9234c049-agent-certs\") pod \"konnectivity-agent-25n4b\" (UID: \"1fa00627-119a-42c6-aff3-568d9234c049\") " pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:20:40.800594 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-socket-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.800594 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-sys-fs\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.800594 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800584    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-socket-dir-parent\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800609    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800629    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-system-cni-dir\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800655    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-run-k8s-cni-cncf-io\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800675    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-socket-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-var-lib-cni-multus\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800705    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqv\" (UniqueName: \"kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv\") pod \"network-check-target-bm5dj\" (UID: \"edb74921-50a4-402a-b2cc-138778d94d25\") " pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800731    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-sys-fs\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-cni-dir\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800765    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-hostroot\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-cni-dir\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-conf-dir\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-daemon-config\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800820    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-conf-dir\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800843    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f92dc\" (UniqueName: \"kubernetes.io/projected/cc34fc3b-3dee-4485-87ba-988209781305-kube-api-access-f92dc\") pod \"node-ca-jnctc\" (UID: \"cc34fc3b-3dee-4485-87ba-988209781305\") " pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.800850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800844    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc34fc3b-3dee-4485-87ba-988209781305-serviceca\") pod \"node-ca-jnctc\" (UID: \"cc34fc3b-3dee-4485-87ba-988209781305\") " pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800860    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-hostroot\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800861    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-system-cni-dir\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800880    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1fa00627-119a-42c6-aff3-568d9234c049-konnectivity-ca\") pod \"konnectivity-agent-25n4b\" (UID: \"1fa00627-119a-42c6-aff3-568d9234c049\") " pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800888    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-socket-dir-parent\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800897    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-var-lib-cni-multus\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800893    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-run-k8s-cni-cncf-io\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-device-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800935    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-run-multus-certs\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.800967    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.800988    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62a9d73d-1877-48b9-94e2-5a76311b7453-kubelet-config\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62a9d73d-1877-48b9-94e2-5a76311b7453-dbus\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.801028    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs podName:4e506cbe-bfeb-4614-b6c7-91b349785053 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:41.301008676 +0000 UTC m=+4.794361245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs") pod "network-metrics-daemon-7kdrq" (UID: "4e506cbe-bfeb-4614-b6c7-91b349785053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801062    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-cnibin\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801079    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-device-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.801605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801131    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62a9d73d-1877-48b9-94e2-5a76311b7453-dbus\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801135    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4fbb\" (UniqueName: \"kubernetes.io/projected/055c8e5b-8db0-441b-a338-08f6572fe0ae-kube-api-access-z4fbb\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801181    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-cnibin\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801162    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37093cb5-2840-4b9c-97b6-f66bbd717c9c-cni-binary-copy\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801212    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-run-netns\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801231    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.801242    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801237    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-etc-kubernetes\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801283    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-run-multus-certs\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801284    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37093cb5-2840-4b9c-97b6-f66bbd717c9c-multus-daemon-config\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.801288    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret podName:62a9d73d-1877-48b9-94e2-5a76311b7453 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:41.301273869 +0000 UTC m=+4.794626425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret") pod "global-pull-secret-syncer-l6x2p" (UID: "62a9d73d-1877-48b9-94e2-5a76311b7453") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801328    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-etc-kubernetes\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801335    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37093cb5-2840-4b9c-97b6-f66bbd717c9c-host-run-netns\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc34fc3b-3dee-4485-87ba-988209781305-host\") pod \"node-ca-jnctc\" (UID: \"cc34fc3b-3dee-4485-87ba-988209781305\") " pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801352    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62a9d73d-1877-48b9-94e2-5a76311b7453-kubelet-config\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc34fc3b-3dee-4485-87ba-988209781305-host\") pod \"node-ca-jnctc\" (UID: \"cc34fc3b-3dee-4485-87ba-988209781305\") " pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801371    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl8mt\" (UniqueName: \"kubernetes.io/projected/4e506cbe-bfeb-4614-b6c7-91b349785053-kube-api-access-bl8mt\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:40.802265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801402    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw26h\" (UniqueName: \"kubernetes.io/projected/37093cb5-2840-4b9c-97b6-f66bbd717c9c-kube-api-access-jw26h\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.802847 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-registration-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.802847 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801447    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1fa00627-119a-42c6-aff3-568d9234c049-konnectivity-ca\") pod \"konnectivity-agent-25n4b\" (UID: \"1fa00627-119a-42c6-aff3-568d9234c049\") " pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:20:40.802847 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801516    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/055c8e5b-8db0-441b-a338-08f6572fe0ae-registration-dir\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.802847 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.801681    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37093cb5-2840-4b9c-97b6-f66bbd717c9c-cni-binary-copy\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.803380 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.803365    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1fa00627-119a-42c6-aff3-568d9234c049-agent-certs\") pod \"konnectivity-agent-25n4b\" (UID: \"1fa00627-119a-42c6-aff3-568d9234c049\") " pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:20:40.812951 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.812920    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:40.812951 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.812946    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:40.813104 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.812960    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqv for pod openshift-network-diagnostics/network-check-target-bm5dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:40.813104 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:40.813018    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv podName:edb74921-50a4-402a-b2cc-138778d94d25 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:41.313003132 +0000 UTC m=+4.806355688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqv" (UniqueName: "kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv") pod "network-check-target-bm5dj" (UID: "edb74921-50a4-402a-b2cc-138778d94d25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:40.815265 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.815235    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f92dc\" (UniqueName: \"kubernetes.io/projected/cc34fc3b-3dee-4485-87ba-988209781305-kube-api-access-f92dc\") pod \"node-ca-jnctc\" (UID: \"cc34fc3b-3dee-4485-87ba-988209781305\") " pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.815374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.815358    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw26h\" (UniqueName: \"kubernetes.io/projected/37093cb5-2840-4b9c-97b6-f66bbd717c9c-kube-api-access-jw26h\") pod \"multus-b6xk6\" (UID: \"37093cb5-2840-4b9c-97b6-f66bbd717c9c\") " pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:40.815631 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.815609    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4fbb\" (UniqueName: \"kubernetes.io/projected/055c8e5b-8db0-441b-a338-08f6572fe0ae-kube-api-access-z4fbb\") pod \"aws-ebs-csi-driver-node-sbc2k\" (UID: \"055c8e5b-8db0-441b-a338-08f6572fe0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.815631 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.815628    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl8mt\" (UniqueName: \"kubernetes.io/projected/4e506cbe-bfeb-4614-b6c7-91b349785053-kube-api-access-bl8mt\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:40.891341 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.891275    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v2ks2"
Sep 18 20:20:40.905245 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.905225    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zlggb"
Sep 18 20:20:40.914130 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.914112    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w2rft"
Sep 18 20:20:40.920984 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.920967    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hl4v9"
Sep 18 20:20:40.927710 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.927679    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:20:40.936656 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.936633    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k"
Sep 18 20:20:40.942747 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.942720    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:20:40.950348 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.950333    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jnctc"
Sep 18 20:20:40.957201 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:40.957187    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b6xk6"
Sep 18 20:20:41.304497 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.304406    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:41.304649 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.304508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:41.304649 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:41.304587    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:41.304754 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:41.304648    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:41.304754 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:41.304673    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs podName:4e506cbe-bfeb-4614-b6c7-91b349785053 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:42.304653561 +0000 UTC m=+5.798006134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs") pod "network-metrics-daemon-7kdrq" (UID: "4e506cbe-bfeb-4614-b6c7-91b349785053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:41.304754 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:41.304697    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret podName:62a9d73d-1877-48b9-94e2-5a76311b7453 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:42.304682189 +0000 UTC m=+5.798034751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret") pod "global-pull-secret-syncer-l6x2p" (UID: "62a9d73d-1877-48b9-94e2-5a76311b7453") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:41.405033 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.405002    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqv\" (UniqueName: \"kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv\") pod \"network-check-target-bm5dj\" (UID: \"edb74921-50a4-402a-b2cc-138778d94d25\") " pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:41.405201 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:41.405160    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:41.405201 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:41.405182    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:41.405201 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:41.405193    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqv for pod openshift-network-diagnostics/network-check-target-bm5dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:41.405352 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:41.405255    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv podName:edb74921-50a4-402a-b2cc-138778d94d25 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:42.405236836 +0000 UTC m=+5.898589409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqv" (UniqueName: "kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv") pod "network-check-target-bm5dj" (UID: "edb74921-50a4-402a-b2cc-138778d94d25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:41.557324 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:41.557300    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03a0417_3da2_4804_b145_74d2f4954f2c.slice/crio-3cedd2edc5b1859327728a1c41b4fc140719d8b6511b936c328be53499d43ea5 WatchSource:0}: Error finding container 3cedd2edc5b1859327728a1c41b4fc140719d8b6511b936c328be53499d43ea5: Status 404 returned error can't find the container with id 3cedd2edc5b1859327728a1c41b4fc140719d8b6511b936c328be53499d43ea5
Sep 18 20:20:41.558839 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:41.558819    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79598812_762d_49dc_94e2_91f9525c69a9.slice/crio-e4592087421e9ff91298680426fdc66a5a4698246efad3af4d43d8393166dfef WatchSource:0}: Error finding container e4592087421e9ff91298680426fdc66a5a4698246efad3af4d43d8393166dfef: Status 404 returned error can't find the container with id e4592087421e9ff91298680426fdc66a5a4698246efad3af4d43d8393166dfef
Sep 18 20:20:41.559571 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:41.559545    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2d5d99_7944_4e3c_95f7_a849c81acda1.slice/crio-a5c24806d858aac7ae12c57145f5352a38c716d23356d4cd9055b722405d6ace WatchSource:0}: Error finding container a5c24806d858aac7ae12c57145f5352a38c716d23356d4cd9055b722405d6ace: Status 404 returned error can't find the container with id a5c24806d858aac7ae12c57145f5352a38c716d23356d4cd9055b722405d6ace
Sep 18 20:20:41.560630 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:41.560447    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c0c73b_2139_4f34_9bc6_a972fb211a9f.slice/crio-02ee67fb1356e3034edd6984acebcb621b768e0e9258102f353676235923044d WatchSource:0}: Error finding container 02ee67fb1356e3034edd6984acebcb621b768e0e9258102f353676235923044d: Status 404 returned error can't find the container with id 02ee67fb1356e3034edd6984acebcb621b768e0e9258102f353676235923044d
Sep 18 20:20:41.563425 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:41.563405    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37093cb5_2840_4b9c_97b6_f66bbd717c9c.slice/crio-35979e4bad800aa09ebc2255b7d7c6988f1991822870af9357000617bd7c8017 WatchSource:0}: Error finding container 35979e4bad800aa09ebc2255b7d7c6988f1991822870af9357000617bd7c8017: Status 404 returned error can't find the container with id 35979e4bad800aa09ebc2255b7d7c6988f1991822870af9357000617bd7c8017
Sep 18 20:20:41.565275 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:41.565194    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055c8e5b_8db0_441b_a338_08f6572fe0ae.slice/crio-ebf34ff5051c064003afda2f05a0639df2c14989358a6cf5e7804278b85e48a4 WatchSource:0}: Error finding container ebf34ff5051c064003afda2f05a0639df2c14989358a6cf5e7804278b85e48a4: Status 404 returned error can't find the container with id ebf34ff5051c064003afda2f05a0639df2c14989358a6cf5e7804278b85e48a4
Sep 18 20:20:41.565565 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:41.565545    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa00627_119a_42c6_aff3_568d9234c049.slice/crio-ed1eff3716b5e8b85d3f4ede2236172e50e82e11110118abda5f9c988e81d17b WatchSource:0}: Error finding container ed1eff3716b5e8b85d3f4ede2236172e50e82e11110118abda5f9c988e81d17b: Status 404 returned error can't find the container with id ed1eff3716b5e8b85d3f4ede2236172e50e82e11110118abda5f9c988e81d17b
Sep 18 20:20:41.567528 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:41.567161    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcceb3e3_7696_404d_a730_edc53df512cc.slice/crio-65410c43b246551132f7c117f53a47d9a5664d42ee8945ee61f6f1589a91ba30 WatchSource:0}: Error finding container 65410c43b246551132f7c117f53a47d9a5664d42ee8945ee61f6f1589a91ba30: Status 404 returned error can't find the container with id 65410c43b246551132f7c117f53a47d9a5664d42ee8945ee61f6f1589a91ba30
Sep 18 20:20:41.567528 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:20:41.567521    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc34fc3b_3dee_4485_87ba_988209781305.slice/crio-aaa0b99d8e72cfbb1335e9b28e369c58e482fed3770747937d0ca86c70294a61 WatchSource:0}: Error finding container aaa0b99d8e72cfbb1335e9b28e369c58e482fed3770747937d0ca86c70294a61: Status 404 returned error can't find the container with id aaa0b99d8e72cfbb1335e9b28e369c58e482fed3770747937d0ca86c70294a61
Sep 18 20:20:41.631066 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.630932    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 20:15:40 +0000 UTC" deadline="2028-05-05 03:26:25.685555108 +0000 UTC"
Sep 18 20:20:41.631066 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.631063    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14263h5m44.054494844s"
Sep 18 20:20:41.716733 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.716705    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:41.716860 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:41.716829    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:41.726591 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.726563    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v2ks2" event={"ID":"fcceb3e3-7696-404d-a730-edc53df512cc","Type":"ContainerStarted","Data":"65410c43b246551132f7c117f53a47d9a5664d42ee8945ee61f6f1589a91ba30"}
Sep 18 20:20:41.727761 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.727740    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zlggb" event={"ID":"86c0c73b-2139-4f34-9bc6-a972fb211a9f","Type":"ContainerStarted","Data":"02ee67fb1356e3034edd6984acebcb621b768e0e9258102f353676235923044d"}
Sep 18 20:20:41.729036 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.729014    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" event={"ID":"c03a0417-3da2-4804-b145-74d2f4954f2c","Type":"ContainerStarted","Data":"3cedd2edc5b1859327728a1c41b4fc140719d8b6511b936c328be53499d43ea5"}
Sep 18 20:20:41.732085 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.732064    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal" event={"ID":"7d826f4c7a124136216df58c6a6c0d1b","Type":"ContainerStarted","Data":"d91f575efda367962742de0b52f52904853eb1b732f5336f3dc73425223c05f8"}
Sep 18 20:20:41.734105 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.734084    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jnctc" event={"ID":"cc34fc3b-3dee-4485-87ba-988209781305","Type":"ContainerStarted","Data":"aaa0b99d8e72cfbb1335e9b28e369c58e482fed3770747937d0ca86c70294a61"}
Sep 18 20:20:41.735176 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.735141    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-25n4b" event={"ID":"1fa00627-119a-42c6-aff3-568d9234c049","Type":"ContainerStarted","Data":"ed1eff3716b5e8b85d3f4ede2236172e50e82e11110118abda5f9c988e81d17b"}
Sep 18 20:20:41.736681 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.736659    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k" event={"ID":"055c8e5b-8db0-441b-a338-08f6572fe0ae","Type":"ContainerStarted","Data":"ebf34ff5051c064003afda2f05a0639df2c14989358a6cf5e7804278b85e48a4"}
Sep 18 20:20:41.738241 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.738221    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b6xk6" event={"ID":"37093cb5-2840-4b9c-97b6-f66bbd717c9c","Type":"ContainerStarted","Data":"35979e4bad800aa09ebc2255b7d7c6988f1991822870af9357000617bd7c8017"}
Sep 18 20:20:41.739251 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.739228    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl4v9" event={"ID":"9b2d5d99-7944-4e3c-95f7-a849c81acda1","Type":"ContainerStarted","Data":"a5c24806d858aac7ae12c57145f5352a38c716d23356d4cd9055b722405d6ace"}
Sep 18 20:20:41.740291 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.740274    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w2rft" event={"ID":"79598812-762d-49dc-94e2-91f9525c69a9","Type":"ContainerStarted","Data":"e4592087421e9ff91298680426fdc66a5a4698246efad3af4d43d8393166dfef"}
Sep 18 20:20:41.748417 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:41.748377    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-28.ec2.internal" podStartSLOduration=2.7483670350000002 podStartE2EDuration="2.748367035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:41.747734033 +0000 UTC m=+5.241086612" watchObservedRunningTime="2026-09-18 20:20:41.748367035 +0000 UTC m=+5.241719612"
Sep 18 20:20:42.094733 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:42.094491    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:42.313598 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:42.312868    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:42.313598 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:42.312939    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:42.313598 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:42.313059    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:42.313598 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:42.313117    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret podName:62a9d73d-1877-48b9-94e2-5a76311b7453 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:44.313098467 +0000 UTC m=+7.806451043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret") pod "global-pull-secret-syncer-l6x2p" (UID: "62a9d73d-1877-48b9-94e2-5a76311b7453") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:42.313598 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:42.313511    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:42.313598 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:42.313562    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs podName:4e506cbe-bfeb-4614-b6c7-91b349785053 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:44.313545089 +0000 UTC m=+7.806897651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs") pod "network-metrics-daemon-7kdrq" (UID: "4e506cbe-bfeb-4614-b6c7-91b349785053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:42.413927 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:42.413240    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqv\" (UniqueName: \"kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv\") pod \"network-check-target-bm5dj\" (UID: \"edb74921-50a4-402a-b2cc-138778d94d25\") " pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:42.413927 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:42.413439    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:42.413927 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:42.413477    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:42.413927 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:42.413490    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqv for pod openshift-network-diagnostics/network-check-target-bm5dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:42.413927 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:42.413544    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv podName:edb74921-50a4-402a-b2cc-138778d94d25 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:44.413526611 +0000 UTC m=+7.906879172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqv" (UniqueName: "kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv") pod "network-check-target-bm5dj" (UID: "edb74921-50a4-402a-b2cc-138778d94d25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:42.634174 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:42.632021    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 20:15:40 +0000 UTC" deadline="2028-04-18 03:00:10.349856129 +0000 UTC"
Sep 18 20:20:42.634174 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:42.632056    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13854h39m27.717804231s"
Sep 18 20:20:42.716848 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:42.716742    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:42.716992 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:42.716886    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:20:42.717242 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:42.716742    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:42.717368 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:42.717337    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:20:42.758162 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:42.758131    2557 generic.go:356] "Generic (PLEG): container finished" podID="ee72dde8fdd947ba26738b5722deca91" containerID="d82581b5ddb237a23767dcb35e513c450ac3ee941f1411a06d7c66251de01165" exitCode=0
Sep 18 20:20:42.758309 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:42.758272    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal" event={"ID":"ee72dde8fdd947ba26738b5722deca91","Type":"ContainerDied","Data":"d82581b5ddb237a23767dcb35e513c450ac3ee941f1411a06d7c66251de01165"}
Sep 18 20:20:43.716772 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:43.716738    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:43.717216 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:43.716870    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:43.764252 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:43.764213    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal" event={"ID":"ee72dde8fdd947ba26738b5722deca91","Type":"ContainerStarted","Data":"972b68e1ac707dc575dada16423a7e5367fd856ec9c872407dfbd036b3431562"}
Sep 18 20:20:44.330446 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:44.330409    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:44.330680 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:44.330500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:44.330680 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:44.330641    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:44.330794 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:44.330701    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs podName:4e506cbe-bfeb-4614-b6c7-91b349785053 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:48.330680172 +0000 UTC m=+11.824032745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs") pod "network-metrics-daemon-7kdrq" (UID: "4e506cbe-bfeb-4614-b6c7-91b349785053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:44.330794 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:44.330771    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:44.330907 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:44.330803    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret podName:62a9d73d-1877-48b9-94e2-5a76311b7453 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:48.330792333 +0000 UTC m=+11.824144907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret") pod "global-pull-secret-syncer-l6x2p" (UID: "62a9d73d-1877-48b9-94e2-5a76311b7453") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:44.430965 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:44.430932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqv\" (UniqueName: \"kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv\") pod \"network-check-target-bm5dj\" (UID: \"edb74921-50a4-402a-b2cc-138778d94d25\") " pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:44.431139 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:44.431120    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:44.431139 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:44.431137    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:44.431250 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:44.431150    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqv for pod openshift-network-diagnostics/network-check-target-bm5dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:44.431250 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:44.431205    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv podName:edb74921-50a4-402a-b2cc-138778d94d25 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:48.431188575 +0000 UTC m=+11.924541145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqv" (UniqueName: "kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv") pod "network-check-target-bm5dj" (UID: "edb74921-50a4-402a-b2cc-138778d94d25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:44.719672 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:44.719600    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:44.720163 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:44.719740    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:20:44.720163 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:44.720122    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:44.720272 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:44.720209    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:20:45.716870 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:45.716838    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:45.717050 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:45.716965    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:46.717575 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:46.717538    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:46.718048 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:46.717550    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:46.718048 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:46.717672    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:20:46.718048 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:46.717792    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:20:47.717553 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:47.717512    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:47.717724 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:47.717657    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:48.366113 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:48.366078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:48.366294 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:48.366165    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:48.366294 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:48.366260    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:48.366294 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:48.366285    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:48.366444 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:48.366334    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret podName:62a9d73d-1877-48b9-94e2-5a76311b7453 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:56.366310125 +0000 UTC m=+19.859662697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret") pod "global-pull-secret-syncer-l6x2p" (UID: "62a9d73d-1877-48b9-94e2-5a76311b7453") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:48.366444 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:48.366355    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs podName:4e506cbe-bfeb-4614-b6c7-91b349785053 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:56.36634422 +0000 UTC m=+19.859696782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs") pod "network-metrics-daemon-7kdrq" (UID: "4e506cbe-bfeb-4614-b6c7-91b349785053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:48.467664 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:48.467018    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqv\" (UniqueName: \"kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv\") pod \"network-check-target-bm5dj\" (UID: \"edb74921-50a4-402a-b2cc-138778d94d25\") " pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:48.467664 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:48.467233    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:48.467664 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:48.467256    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:48.467664 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:48.467268    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqv for pod openshift-network-diagnostics/network-check-target-bm5dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:48.467664 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:48.467320    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv podName:edb74921-50a4-402a-b2cc-138778d94d25 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:56.46730232 +0000 UTC m=+19.960654895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqv" (UniqueName: "kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv") pod "network-check-target-bm5dj" (UID: "edb74921-50a4-402a-b2cc-138778d94d25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:48.718565 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:48.717912    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:48.718565 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:48.718026    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:20:48.718565 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:48.718377    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:48.718565 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:48.718477    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:20:49.716868 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:49.716828    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:49.717026 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:49.716957    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:50.716828 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:50.716798    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:50.717318 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:50.716802    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:50.717318 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:50.716917    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:20:50.717318 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:50.717008    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:20:51.716940 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:51.716865    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:51.717385 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:51.716989    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:52.717180 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:52.717146    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:52.717661 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:52.717284    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:20:52.717661 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:52.717345    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:52.717661 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:52.717470    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:20:53.717443 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:53.717406    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:53.717945 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:53.717550    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:54.717078 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:54.717042    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:54.717223 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:54.717190    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:20:54.717274 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:54.717246    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:54.717395 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:54.717344    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:20:55.717630 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:55.717597    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:55.718038 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:55.717732    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:56.427314 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:56.427275    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:56.427503 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:56.427357    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:56.427503 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:56.427444    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:56.427503 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:56.427484    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:56.427684 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:56.427534    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs podName:4e506cbe-bfeb-4614-b6c7-91b349785053 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:12.427513066 +0000 UTC m=+35.920865626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs") pod "network-metrics-daemon-7kdrq" (UID: "4e506cbe-bfeb-4614-b6c7-91b349785053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:56.427684 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:56.427555    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret podName:62a9d73d-1877-48b9-94e2-5a76311b7453 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:12.427545401 +0000 UTC m=+35.920897963 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret") pod "global-pull-secret-syncer-l6x2p" (UID: "62a9d73d-1877-48b9-94e2-5a76311b7453") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:56.528627 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:56.528593    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqv\" (UniqueName: \"kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv\") pod \"network-check-target-bm5dj\" (UID: \"edb74921-50a4-402a-b2cc-138778d94d25\") " pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:56.528800 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:56.528756    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:56.528800 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:56.528783    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:56.528800 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:56.528796    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqv for pod openshift-network-diagnostics/network-check-target-bm5dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:56.528951 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:56.528852    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv podName:edb74921-50a4-402a-b2cc-138778d94d25 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:12.528831183 +0000 UTC m=+36.022183762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqv" (UniqueName: "kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv") pod "network-check-target-bm5dj" (UID: "edb74921-50a4-402a-b2cc-138778d94d25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:56.716839 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:56.716807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:56.716995 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:56.716928    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:20:56.717076 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:56.717008    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:56.717142 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:56.717119    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:20:57.717646 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:57.717610    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:57.718039 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:57.717741    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:58.717881 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:58.717849    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:20:58.718283 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:58.717957    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:20:58.718283 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:58.718013    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:20:58.718283 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:58.718106    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:20:59.717811 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.717573    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:20:59.717911 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:20:59.717890    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:20:59.794882 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.794853    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v2ks2" event={"ID":"fcceb3e3-7696-404d-a730-edc53df512cc","Type":"ContainerStarted","Data":"3dc60d4a9f7536ada30cc2b99fe4461e331d2aff2a68a15263dafcfe1b07f2cd"}
Sep 18 20:20:59.798201 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.798177    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:20:59.798648 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.798620    2557 generic.go:356] "Generic (PLEG): container finished" podID="c03a0417-3da2-4804-b145-74d2f4954f2c" containerID="8a377a0faf3664e86070a32d676c0751b1924183668d74ca6240f9868e3f1074" exitCode=1
Sep 18 20:20:59.798731 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.798684    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" event={"ID":"c03a0417-3da2-4804-b145-74d2f4954f2c","Type":"ContainerStarted","Data":"90615aa64b0de8ef779a80f8a33d2d8492a8e077a7f9eec059a36fa6661f9db7"}
Sep 18 20:20:59.798776 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.798756    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" event={"ID":"c03a0417-3da2-4804-b145-74d2f4954f2c","Type":"ContainerStarted","Data":"1d30130f61b10b3b1af7ab7f64dedb3343d9b26486b1c68838fcab5563dadc9e"}
Sep 18 20:20:59.798829 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.798809    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" event={"ID":"c03a0417-3da2-4804-b145-74d2f4954f2c","Type":"ContainerStarted","Data":"5ddbe51bedb381b872c0669458c01c98db155176f275a1af8e604a4f8ecde1bc"}
Sep 18 20:20:59.798898 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.798884    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" event={"ID":"c03a0417-3da2-4804-b145-74d2f4954f2c","Type":"ContainerStarted","Data":"90fe77d26b1096ae69b59e861daa30cc3601a003e71064d82de47b988c2b5fc9"}
Sep 18 20:20:59.798947 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.798935    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" event={"ID":"c03a0417-3da2-4804-b145-74d2f4954f2c","Type":"ContainerDied","Data":"8a377a0faf3664e86070a32d676c0751b1924183668d74ca6240f9868e3f1074"}
Sep 18 20:20:59.798998 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.798985    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" event={"ID":"c03a0417-3da2-4804-b145-74d2f4954f2c","Type":"ContainerStarted","Data":"4e9a6c06bdc49461096ffc5b926700e8a47a175d67c4204543bb99340be747cb"}
Sep 18 20:20:59.800467 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.800433    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jnctc" event={"ID":"cc34fc3b-3dee-4485-87ba-988209781305","Type":"ContainerStarted","Data":"7862e50e265009cf7f175e54e0d9a4094c37b49bc31ce35f5a860d6bbd902c9f"}
Sep 18 20:20:59.802242 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.802216    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-25n4b" event={"ID":"1fa00627-119a-42c6-aff3-568d9234c049","Type":"ContainerStarted","Data":"42fbbfda8b866a571bb1ccc90bf8c7e26b1e8512c33fab13b7b8aead33caf505"}
Sep 18 20:20:59.804068 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.804041    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k" event={"ID":"055c8e5b-8db0-441b-a338-08f6572fe0ae","Type":"ContainerStarted","Data":"b802c6ea468667b7272f849f3e67289c9703d0c39cfdfcde6d05bb9db318f4fc"}
Sep 18 20:20:59.805661 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.805635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b6xk6" event={"ID":"37093cb5-2840-4b9c-97b6-f66bbd717c9c","Type":"ContainerStarted","Data":"fec8e8de87b4c7248d48c5bc6794f29eb007cb84637b4aaa68016be5110d66ad"}
Sep 18 20:20:59.810309 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.810264    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v2ks2" podStartSLOduration=4.582479597 podStartE2EDuration="21.810240542s" totalImagesPullingTime="17.227760945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:59.809955766 +0000 UTC m=+23.303308346" watchObservedRunningTime="2026-09-18 20:20:59.810240542 +0000 UTC m=+23.303593114"
Sep 18 20:20:59.810505 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.810472    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-28.ec2.internal" podStartSLOduration=20.810442695 podStartE2EDuration="20.810442695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:43.780191413 +0000 UTC m=+7.273544195" watchObservedRunningTime="2026-09-18 20:20:59.810442695 +0000 UTC m=+23.303795273"
Sep 18 20:20:59.810904 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.810867    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b2d5d99-7944-4e3c-95f7-a849c81acda1" containerID="bb67fcc498061f0c5a08a7279f8b5230ff57aa4fd9c1dadff14960fd39648162" exitCode=0
Sep 18 20:20:59.810994 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.810922    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl4v9" event={"ID":"9b2d5d99-7944-4e3c-95f7-a849c81acda1","Type":"ContainerDied","Data":"bb67fcc498061f0c5a08a7279f8b5230ff57aa4fd9c1dadff14960fd39648162"}
Sep 18 20:20:59.813000 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.812972    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w2rft" event={"ID":"79598812-762d-49dc-94e2-91f9525c69a9","Type":"ContainerStarted","Data":"0d17e617b9676f942276b099d53dd2427a61a74263e8dce23766635d41b7e780"}
Sep 18 20:20:59.825758 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.825720    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b6xk6" podStartSLOduration=4.418741591 podStartE2EDuration="21.825705691s" totalImagesPullingTime="17.4069641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:59.825338694 +0000 UTC m=+23.318691271" watchObservedRunningTime="2026-09-18 20:20:59.825705691 +0000 UTC m=+23.319058270"
Sep 18 20:20:59.839623 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.839590    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-25n4b" podStartSLOduration=8.739649515 podStartE2EDuration="21.839577641s" totalImagesPullingTime="13.099928126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:59.839056551 +0000 UTC m=+23.332409129" watchObservedRunningTime="2026-09-18 20:20:59.839577641 +0000 UTC m=+23.332930220"
Sep 18 20:20:59.853227 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.853193    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jnctc" podStartSLOduration=4.509955008 podStartE2EDuration="21.853178066s" totalImagesPullingTime="17.343223058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:59.852664905 +0000 UTC m=+23.346017482" watchObservedRunningTime="2026-09-18 20:20:59.853178066 +0000 UTC m=+23.346530645"
Sep 18 20:20:59.869277 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.869236    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w2rft" podStartSLOduration=4.515351652 podStartE2EDuration="21.869222206s" totalImagesPullingTime="17.353870554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:59.868863465 +0000 UTC m=+23.362216048" watchObservedRunningTime="2026-09-18 20:20:59.869222206 +0000 UTC m=+23.362574830"
Sep 18 20:20:59.950687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:20:59.950666    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:00.016917 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.016888    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:00.165081 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.165052    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 20:21:00.596665 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.596629    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:00.668295 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.668269    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:00.695809 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.695700    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T20:21:00.165076035Z","UUID":"ae09abd7-72c1-4ff4-acaf-92b01f986389","Handler":null,"Name":"","Endpoint":""}
Sep 18 20:21:00.698342 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.698310    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 20:21:00.698342 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.698347    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 20:21:00.716728 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.716704    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:00.716812 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.716707    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:00.716847 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:00.716819    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:21:00.716923 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:00.716886    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:21:00.818123 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.818088    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k" event={"ID":"055c8e5b-8db0-441b-a338-08f6572fe0ae","Type":"ContainerStarted","Data":"8140f15bcb282ff75a9d470aeacf801202166775214d211e8f73b8d5101e8879"}
Sep 18 20:21:00.819857 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.819825    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zlggb" event={"ID":"86c0c73b-2139-4f34-9bc6-a972fb211a9f","Type":"ContainerStarted","Data":"c37d3edb0da881c5757262ce6ade3d5a034e7adc72c5c8db9d932a1808bc2a54"}
Sep 18 20:21:00.955328 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:00.955296    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:01.055504 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:01.055474    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:01.716861 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:01.716690    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:01.717009 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:01.716933    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:21:01.824714 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:01.824684    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k" event={"ID":"055c8e5b-8db0-441b-a338-08f6572fe0ae","Type":"ContainerStarted","Data":"4f9caf31dba8383694965c65bd7bce951453c60966976a278d42d3e39742dcee"}
Sep 18 20:21:01.866608 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:01.866570    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbc2k" podStartSLOduration=4.7428763069999995 podStartE2EDuration="23.866555084s" totalImagesPullingTime="19.123678777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:01.864164346 +0000 UTC m=+25.357516926" watchObservedRunningTime="2026-09-18 20:21:01.866555084 +0000 UTC m=+25.359907673"
Sep 18 20:21:01.866974 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:01.866925    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zlggb" podStartSLOduration=6.515401625 podStartE2EDuration="23.866918044s" totalImagesPullingTime="17.351516419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:00.83399405 +0000 UTC m=+24.327346628" watchObservedRunningTime="2026-09-18 20:21:01.866918044 +0000 UTC m=+25.360270621"
Sep 18 20:21:01.898445 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:01.898417    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:02.023210 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:02.023135    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:02.717214 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:02.717176    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:02.717214 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:02.717191    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:02.717441 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:02.717314    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:21:02.717525 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:02.717473    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:21:02.830753 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:02.830728    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:21:02.831255 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:02.831120    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" event={"ID":"c03a0417-3da2-4804-b145-74d2f4954f2c","Type":"ContainerStarted","Data":"8ea4e96f6c6ecb5a8b805d818621cdbb089fab1250fb385771ff99f937b0e3d7"}
Sep 18 20:21:03.595795 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:03.595761    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:03.675446 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:03.675405    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:03.716952 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:03.716926    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:03.717121 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:03.717042    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:21:04.188993 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.188806    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:21:04.189609 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.189300    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:21:04.409791 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.409765    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:21:04.410165 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.410152    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-25n4b"
Sep 18 20:21:04.469908 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.469856    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:04.717559 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.717534    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:04.717688 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:04.717656    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:21:04.717746 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.717690    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:04.717800 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:04.717763    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:21:04.836887 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.836831    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b2d5d99-7944-4e3c-95f7-a849c81acda1" containerID="1db17bde3befe4d396c0b6cf13d1b6e353e52b88de9ad94f268d298fce8695cf" exitCode=0
Sep 18 20:21:04.836976 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.836919    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl4v9" event={"ID":"9b2d5d99-7944-4e3c-95f7-a849c81acda1","Type":"ContainerDied","Data":"1db17bde3befe4d396c0b6cf13d1b6e353e52b88de9ad94f268d298fce8695cf"}
Sep 18 20:21:04.840639 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.840620    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:21:04.841035 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.841010    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" event={"ID":"c03a0417-3da2-4804-b145-74d2f4954f2c","Type":"ContainerStarted","Data":"45b59fbe9aede6f436fda032fcc003154459aec3809ced273b1ffbf6a9471a1d"}
Sep 18 20:21:04.841526 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:04.841509    2557 scope.go:117] "RemoveContainer" containerID="8a377a0faf3664e86070a32d676c0751b1924183668d74ca6240f9868e3f1074"
Sep 18 20:21:05.717317 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.717290    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:05.717649 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:05.717404    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:21:05.832256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.832227    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bm5dj"]
Sep 18 20:21:05.832392 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.832327    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:05.832487 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:05.832412    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:21:05.833155 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.833125    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7kdrq"]
Sep 18 20:21:05.833247 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.833216    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:05.833363 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:05.833342    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:21:05.833715 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.833696    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l6x2p"]
Sep 18 20:21:05.847418 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.847373    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:21:05.847884 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.847862    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:05.847884 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.847869    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" event={"ID":"c03a0417-3da2-4804-b145-74d2f4954f2c","Type":"ContainerStarted","Data":"1e02401fe4b5e5907caba31a0825c036cda0a5d7d844078bb9e9d22fefff40cb"}
Sep 18 20:21:05.848028 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:05.847943    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:21:05.848181 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.848158    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:21:05.848289 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.848232    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:21:05.848446 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.848421    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:21:05.863512 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.863489    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:21:05.863827 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.863812    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:21:05.875109 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:05.875073    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2" podStartSLOduration=10.513328142 podStartE2EDuration="27.875056729s" totalImagesPullingTime="17.361728587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:05.874822227 +0000 UTC m=+29.368174805" watchObservedRunningTime="2026-09-18 20:21:05.875056729 +0000 UTC m=+29.368409309"
Sep 18 20:21:06.851505 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:06.851478    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b2d5d99-7944-4e3c-95f7-a849c81acda1" containerID="389fc240115e42ae83c533a73ddb54dd1fa8085eef96efd4ae78e742a33442e3" exitCode=0
Sep 18 20:21:06.851863 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:06.851560    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl4v9" event={"ID":"9b2d5d99-7944-4e3c-95f7-a849c81acda1","Type":"ContainerDied","Data":"389fc240115e42ae83c533a73ddb54dd1fa8085eef96efd4ae78e742a33442e3"}
Sep 18 20:21:07.717213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:07.717180    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:07.717392 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:07.717180    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:07.717392 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:07.717280    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:21:07.717392 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:07.717180    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:07.717561 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:07.717354    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:21:07.717561 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:07.717448    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:21:07.803270 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:07.803242    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:07.885831 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:07.885793    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:08.162097 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:08.162066    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:08.241044 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:08.240985    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:08.590547 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:08.590475    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:08.689344 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:08.689314    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:08.857425 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:08.857358    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b2d5d99-7944-4e3c-95f7-a849c81acda1" containerID="ec7645bf3656061727d9995842a17b45c10647777ce88273f74c22dedc003fb3" exitCode=0
Sep 18 20:21:08.857425 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:08.857397    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl4v9" event={"ID":"9b2d5d99-7944-4e3c-95f7-a849c81acda1","Type":"ContainerDied","Data":"ec7645bf3656061727d9995842a17b45c10647777ce88273f74c22dedc003fb3"}
Sep 18 20:21:08.880706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:08.880678    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:09.008140 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:09.008116    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:09.118940 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:09.118880    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:09.217231 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:09.217212    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:09.368100 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:09.368077    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v2ks2_fcceb3e3-7696-404d-a730-edc53df512cc/dns-node-resolver/0.log"
Sep 18 20:21:09.717083 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:09.717057    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:09.717240 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:09.717057    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:09.717240 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:09.717166    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:21:09.717351 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:09.717259    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:21:09.717351 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:09.717057    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:09.717436 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:09.717359    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:21:10.378390 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:10.378360    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jnctc_cc34fc3b-3dee-4485-87ba-988209781305/node-ca/0.log"
Sep 18 20:21:10.500472 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:10.500411    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:10.572169 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:10.572138    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:11.511156 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:11.511100    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:11.717081 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:11.717056    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:11.717231 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:11.717114    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:11.717231 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:11.717211    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:11.717326 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:11.717236    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kdrq" podUID="4e506cbe-bfeb-4614-b6c7-91b349785053"
Sep 18 20:21:11.717326 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:11.717292    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6x2p" podUID="62a9d73d-1877-48b9-94e2-5a76311b7453"
Sep 18 20:21:11.717431 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:11.717371    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bm5dj" podUID="edb74921-50a4-402a-b2cc-138778d94d25"
Sep 18 20:21:11.912647 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:11.912581    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-28.ec2.internal" event="NodeReady"
Sep 18 20:21:11.912783 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:11.912692    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 20:21:12.447321 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:12.447284    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:12.447621 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:12.447355    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:12.447621 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:12.447450    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:21:12.447621 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:12.447544    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs podName:4e506cbe-bfeb-4614-b6c7-91b349785053 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:44.447525507 +0000 UTC m=+67.940878083 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs") pod "network-metrics-daemon-7kdrq" (UID: "4e506cbe-bfeb-4614-b6c7-91b349785053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:21:12.447621 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:12.447449    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:21:12.447621 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:12.447611    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret podName:62a9d73d-1877-48b9-94e2-5a76311b7453 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:44.447597205 +0000 UTC m=+67.940949778 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret") pod "global-pull-secret-syncer-l6x2p" (UID: "62a9d73d-1877-48b9-94e2-5a76311b7453") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:21:12.548366 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:12.548325    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqv\" (UniqueName: \"kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv\") pod \"network-check-target-bm5dj\" (UID: \"edb74921-50a4-402a-b2cc-138778d94d25\") " pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:12.548756 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:12.548535    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:21:12.548756 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:12.548563    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:21:12.548756 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:12.548576    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqv for pod openshift-network-diagnostics/network-check-target-bm5dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:21:12.548756 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:12.548641    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv podName:edb74921-50a4-402a-b2cc-138778d94d25 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:44.54862065 +0000 UTC m=+68.041973221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqv" (UniqueName: "kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv") pod "network-check-target-bm5dj" (UID: "edb74921-50a4-402a-b2cc-138778d94d25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:21:13.717362 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.717331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:13.717773 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.717426    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:13.717773 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.717475    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:13.719750 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.719734    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 20:21:13.719812 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.719803    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 20:21:13.720214 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.720201    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r5tl8\""
Sep 18 20:21:13.720378 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.720364    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnbxc\""
Sep 18 20:21:13.720448 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.720393    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 20:21:13.720448 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.720394    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 20:21:13.872047 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.872021    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b2d5d99-7944-4e3c-95f7-a849c81acda1" containerID="60234611e2f998d5e1f7ddcf8b7991092852972020251dc5b6cfa7107215976d" exitCode=0
Sep 18 20:21:13.872142 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:13.872078    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl4v9" event={"ID":"9b2d5d99-7944-4e3c-95f7-a849c81acda1","Type":"ContainerDied","Data":"60234611e2f998d5e1f7ddcf8b7991092852972020251dc5b6cfa7107215976d"}
Sep 18 20:21:14.876793 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:14.876759    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b2d5d99-7944-4e3c-95f7-a849c81acda1" containerID="32d862993db6ebaf77a33377cd3efb62ff22b493dada2d6753482e4a2cbcab3a" exitCode=0
Sep 18 20:21:14.877161 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:14.876801    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl4v9" event={"ID":"9b2d5d99-7944-4e3c-95f7-a849c81acda1","Type":"ContainerDied","Data":"32d862993db6ebaf77a33377cd3efb62ff22b493dada2d6753482e4a2cbcab3a"}
Sep 18 20:21:14.913527 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:14.913342    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:15.076358 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:15.076329    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:15.188608 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:15.188573    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:15.299013 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:15.298984    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:15.419771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:15.419747    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:15.696375 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:15.696350    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:15.882596 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:15.882568    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl4v9" event={"ID":"9b2d5d99-7944-4e3c-95f7-a849c81acda1","Type":"ContainerStarted","Data":"62f4889f6cfca65f50831f5b82517517825ba1c7b37a61e975d449ed3473fdfa"}
Sep 18 20:21:15.921543 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:15.921502    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hl4v9" podStartSLOduration=10.186730584 podStartE2EDuration="37.921486822s" totalImagesPullingTime="27.734756238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:15.918063532 +0000 UTC m=+39.411416107" watchObservedRunningTime="2026-09-18 20:21:15.921486822 +0000 UTC m=+39.414839399"
Sep 18 20:21:16.458628 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:16.458600    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:16.562162 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:16.562128    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:16.660624 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:16.660594    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:17.035792 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:17.035765    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:17.231221 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:17.231193    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:17.445761 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:17.445698    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:17.626393 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:17.626338    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:18.739710 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:18.739680    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:18.839033 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:18.839005    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:18.992857 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:18.992785    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:19.206313 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:19.206283    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:19.403607 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:19.403544    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:19.599835 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:19.599806    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:20.800767 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:20.800709    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:20.908349 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:20.908327    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:21.008399 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:21.008375    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:21.193491 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:21.193463    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:21.395936 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:21.395909    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:21.611247 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:21.611183    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:22.683471 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:22.683424    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:22.790845 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:22.790817    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:22.990900 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:22.990820    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:23.116124 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:23.116091    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:23.216957 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:23.216925    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:23.398680 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:23.398610    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:23.589960 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:23.589933    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:24.791978 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:24.791950    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:24.997003 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:24.996974    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:25.203317 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:25.203257    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:25.355146 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:25.355118    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:25.453779 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:25.453723    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:25.560050 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:25.560023    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:25.654868 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:25.654840    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:25.842280 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:25.842203    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:26.052828 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:26.052795    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:26.223512 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:26.223489    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:26.427468 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:26.427426    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:26.493575 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:26.493526    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:26.844962 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:26.844892    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:27.033280 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:27.033251    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:27.683447 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:27.683419    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:28.451327 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:28.451295    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:29.431726 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:29.431695    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:29.500055 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:29.500026    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:29.586680 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:29.586652    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:29.725681 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:29.725609    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:30.595423 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:30.595393    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:30.654898 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:30.654871    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:32.886418 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.886391    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf"]
Sep 18 20:21:32.921323 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.921284    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf"]
Sep 18 20:21:32.921323 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.921323    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jlz2j"]
Sep 18 20:21:32.921578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.921397    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf"
Sep 18 20:21:32.924347 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.924320    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 20:21:32.924929 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.924911    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-89cvj\""
Sep 18 20:21:32.945058 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.945038    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jlz2j"]
Sep 18 20:21:32.945157 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.945125    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:32.947674 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.947655    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 20:21:32.947791 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.947656    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 20:21:32.947895 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.947881    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 20:21:32.947941 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.947897    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f4bqd\""
Sep 18 20:21:32.953045 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.952986    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 20:21:32.975257 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.975236    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j6v8w"]
Sep 18 20:21:32.990037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.990015    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-9bhvr"]
Sep 18 20:21:32.990135 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.990018    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:32.994192 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.994168    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 20:21:32.994757 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.994739    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 20:21:32.995099 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:32.995079    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q9622\""
Sep 18 20:21:33.008883 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.008864    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7548dd9977-tkk6p"]
Sep 18 20:21:33.008993 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.008980    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-9bhvr"
Sep 18 20:21:33.010978 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.010958    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fcsp8\""
Sep 18 20:21:33.011087 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.011026    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 20:21:33.011142 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.011121    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 20:21:33.025339 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.025322    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j6v8w"]
Sep 18 20:21:33.025418 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.025350    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-9bhvr"]
Sep 18 20:21:33.025418 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.025363    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7548dd9977-tkk6p"]
Sep 18 20:21:33.025525 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.025423    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.027443 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.027426    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 20:21:33.028042 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.028025    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 20:21:33.028125 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.028076    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 20:21:33.028125 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.028099    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wlxzm\""
Sep 18 20:21:33.035079 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.035058    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 20:21:33.036510 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.036488    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:33.077812 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.077794    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mxhpq"]
Sep 18 20:21:33.082268 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.082253    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-metrics-tls\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.082323 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.082275    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-tmp-dir\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.082323 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.082294    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f94eddb8-352a-4eaa-9301-26a28b73b7fe-data-volume\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.082323 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.082313    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7b1188d9-c781-44ce-9fb8-4534f6f1e0e6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-hzclf\" (UID: \"7b1188d9-c781-44ce-9fb8-4534f6f1e0e6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf"
Sep 18 20:21:33.082426 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.082350    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f94eddb8-352a-4eaa-9301-26a28b73b7fe-crio-socket\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.082426 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.082406    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f94eddb8-352a-4eaa-9301-26a28b73b7fe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.082500 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.082426    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-config-volume\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.082500 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.082441    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcg5q\" (UniqueName: \"kubernetes.io/projected/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-kube-api-access-hcg5q\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.082500 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.082484    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c768v\" (UniqueName: \"kubernetes.io/projected/f94eddb8-352a-4eaa-9301-26a28b73b7fe-kube-api-access-c768v\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.082589 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.082547    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f94eddb8-352a-4eaa-9301-26a28b73b7fe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.098501 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.098481    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mxhpq"]
Sep 18 20:21:33.098606 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.098564    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mxhpq"
Sep 18 20:21:33.101683 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.101664    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 20:21:33.101937 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.101924    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 20:21:33.101993 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.101945    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jdpfx\""
Sep 18 20:21:33.102106 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.102091    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 20:21:33.146989 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.146936    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:33.183223 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f94eddb8-352a-4eaa-9301-26a28b73b7fe-data-volume\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.183372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183228    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-image-registry-private-configuration\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.183372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183246    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-ca-trust-extracted\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.183372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-installation-pull-secrets\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.183372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183276    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66c77\" (UniqueName: \"kubernetes.io/projected/681c9102-7acf-4450-b2b7-2a65aac11518-kube-api-access-66c77\") pod \"downloads-6594f4c75-9bhvr\" (UID: \"681c9102-7acf-4450-b2b7-2a65aac11518\") " pod="openshift-console/downloads-6594f4c75-9bhvr"
Sep 18 20:21:33.183372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7b1188d9-c781-44ce-9fb8-4534f6f1e0e6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-hzclf\" (UID: \"7b1188d9-c781-44ce-9fb8-4534f6f1e0e6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf"
Sep 18 20:21:33.183372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f94eddb8-352a-4eaa-9301-26a28b73b7fe-crio-socket\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.183706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183387    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-registry-tls\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.183706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183426    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f94eddb8-352a-4eaa-9301-26a28b73b7fe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.183706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183471    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-bound-sa-token\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.183706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183506    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f94eddb8-352a-4eaa-9301-26a28b73b7fe-crio-socket\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.183706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183511    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f94eddb8-352a-4eaa-9301-26a28b73b7fe-data-volume\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.183706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183538    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-config-volume\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.183706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183570    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcg5q\" (UniqueName: \"kubernetes.io/projected/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-kube-api-access-hcg5q\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.183706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183610    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c768v\" (UniqueName: \"kubernetes.io/projected/f94eddb8-352a-4eaa-9301-26a28b73b7fe-kube-api-access-c768v\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.183706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-trusted-ca\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.184156 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183725    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f94eddb8-352a-4eaa-9301-26a28b73b7fe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.184156 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183750    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-registry-certificates\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.184156 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcrw\" (UniqueName: \"kubernetes.io/projected/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-kube-api-access-kfcrw\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.184156 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183834    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-metrics-tls\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.184156 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-tmp-dir\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.184156 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.183969    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f94eddb8-352a-4eaa-9301-26a28b73b7fe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.184156 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.184103    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-tmp-dir\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.184516 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.184163    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-config-volume\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.187215 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.187194    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-metrics-tls\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.187351 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.187333    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f94eddb8-352a-4eaa-9301-26a28b73b7fe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.187407 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.187342    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7b1188d9-c781-44ce-9fb8-4534f6f1e0e6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-hzclf\" (UID: \"7b1188d9-c781-44ce-9fb8-4534f6f1e0e6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf"
Sep 18 20:21:33.192321 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.192303    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcg5q\" (UniqueName: \"kubernetes.io/projected/d3a5a620-20ab-44e2-a5dd-89b7c7969dcf-kube-api-access-hcg5q\") pod \"dns-default-j6v8w\" (UID: \"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf\") " pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.195416 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.195390    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c768v\" (UniqueName: \"kubernetes.io/projected/f94eddb8-352a-4eaa-9301-26a28b73b7fe-kube-api-access-c768v\") pod \"insights-runtime-extractor-jlz2j\" (UID: \"f94eddb8-352a-4eaa-9301-26a28b73b7fe\") " pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.229437 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.229415    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf"
Sep 18 20:21:33.241897 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.241872    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:33.252820 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.252800    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jlz2j"
Sep 18 20:21:33.284648 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-image-registry-private-configuration\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.284774 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-ca-trust-extracted\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.284774 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284671    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-installation-pull-secrets\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.284774 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284688    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66c77\" (UniqueName: \"kubernetes.io/projected/681c9102-7acf-4450-b2b7-2a65aac11518-kube-api-access-66c77\") pod \"downloads-6594f4c75-9bhvr\" (UID: \"681c9102-7acf-4450-b2b7-2a65aac11518\") " pod="openshift-console/downloads-6594f4c75-9bhvr"
Sep 18 20:21:33.284774 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284719    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-registry-tls\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.284774 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284750    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-bound-sa-token\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.285039 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284781    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34aed437-eb0e-42d7-94fe-4829714cecc6-cert\") pod \"ingress-canary-mxhpq\" (UID: \"34aed437-eb0e-42d7-94fe-4829714cecc6\") " pod="openshift-ingress-canary/ingress-canary-mxhpq"
Sep 18 20:21:33.285039 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284829    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-trusted-ca\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.285039 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284858    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mncw4\" (UniqueName: \"kubernetes.io/projected/34aed437-eb0e-42d7-94fe-4829714cecc6-kube-api-access-mncw4\") pod \"ingress-canary-mxhpq\" (UID: \"34aed437-eb0e-42d7-94fe-4829714cecc6\") " pod="openshift-ingress-canary/ingress-canary-mxhpq"
Sep 18 20:21:33.285039 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284906    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-registry-certificates\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.285039 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.284931    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcrw\" (UniqueName: \"kubernetes.io/projected/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-kube-api-access-kfcrw\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.285297 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.285046    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-ca-trust-extracted\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.286111 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.286081    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-trusted-ca\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.286217 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.286191    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-registry-certificates\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.287304 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.287280    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-image-registry-private-configuration\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.287404 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.287389    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-installation-pull-secrets\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.288115 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.288100    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-registry-tls\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.297477 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.297227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcrw\" (UniqueName: \"kubernetes.io/projected/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-kube-api-access-kfcrw\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.297477 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.297317    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66c77\" (UniqueName: \"kubernetes.io/projected/681c9102-7acf-4450-b2b7-2a65aac11518-kube-api-access-66c77\") pod \"downloads-6594f4c75-9bhvr\" (UID: \"681c9102-7acf-4450-b2b7-2a65aac11518\") " pod="openshift-console/downloads-6594f4c75-9bhvr"
Sep 18 20:21:33.298114 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.298092    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:33.298713 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.298688    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cd1cf35-e51f-44ec-84fb-d0e22477bbe9-bound-sa-token\") pod \"image-registry-7548dd9977-tkk6p\" (UID: \"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9\") " pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.317060 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.317037    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-9bhvr"
Sep 18 20:21:33.337732 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.337693    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.342109 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.342081    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:33.385836 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.385765    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34aed437-eb0e-42d7-94fe-4829714cecc6-cert\") pod \"ingress-canary-mxhpq\" (UID: \"34aed437-eb0e-42d7-94fe-4829714cecc6\") " pod="openshift-ingress-canary/ingress-canary-mxhpq"
Sep 18 20:21:33.385836 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.385825    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mncw4\" (UniqueName: \"kubernetes.io/projected/34aed437-eb0e-42d7-94fe-4829714cecc6-kube-api-access-mncw4\") pod \"ingress-canary-mxhpq\" (UID: \"34aed437-eb0e-42d7-94fe-4829714cecc6\") " pod="openshift-ingress-canary/ingress-canary-mxhpq"
Sep 18 20:21:33.391427 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.391333    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34aed437-eb0e-42d7-94fe-4829714cecc6-cert\") pod \"ingress-canary-mxhpq\" (UID: \"34aed437-eb0e-42d7-94fe-4829714cecc6\") " pod="openshift-ingress-canary/ingress-canary-mxhpq"
Sep 18 20:21:33.417605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.413369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mncw4\" (UniqueName: \"kubernetes.io/projected/34aed437-eb0e-42d7-94fe-4829714cecc6-kube-api-access-mncw4\") pod \"ingress-canary-mxhpq\" (UID: \"34aed437-eb0e-42d7-94fe-4829714cecc6\") " pod="openshift-ingress-canary/ingress-canary-mxhpq"
Sep 18 20:21:33.552301 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.552273    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf"]
Sep 18 20:21:33.555843 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.555821    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jlz2j"]
Sep 18 20:21:33.556065 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:33.556017    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1188d9_c781_44ce_9fb8_4534f6f1e0e6.slice/crio-229b52045a7f161963dc645e71d1ce8364fd8fe69b30067a4892b63c620628c1 WatchSource:0}: Error finding container 229b52045a7f161963dc645e71d1ce8364fd8fe69b30067a4892b63c620628c1: Status 404 returned error can't find the container with id 229b52045a7f161963dc645e71d1ce8364fd8fe69b30067a4892b63c620628c1
Sep 18 20:21:33.559449 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:33.559427    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94eddb8_352a_4eaa_9301_26a28b73b7fe.slice/crio-2c1fc65ffe686f96557e3a3701af0056b603e2efd2fed5271bcbe1c6a407d3ff WatchSource:0}: Error finding container 2c1fc65ffe686f96557e3a3701af0056b603e2efd2fed5271bcbe1c6a407d3ff: Status 404 returned error can't find the container with id 2c1fc65ffe686f96557e3a3701af0056b603e2efd2fed5271bcbe1c6a407d3ff
Sep 18 20:21:33.612272 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.612251    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j6v8w"]
Sep 18 20:21:33.615918 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:33.615889    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a5a620_20ab_44e2_a5dd_89b7c7969dcf.slice/crio-e21440b834829ad3ff24410220eae028145b49160039300fc347a57f4b52af16 WatchSource:0}: Error finding container e21440b834829ad3ff24410220eae028145b49160039300fc347a57f4b52af16: Status 404 returned error can't find the container with id e21440b834829ad3ff24410220eae028145b49160039300fc347a57f4b52af16
Sep 18 20:21:33.639929 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.639908    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-9bhvr"]
Sep 18 20:21:33.641999 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.641980    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7548dd9977-tkk6p"]
Sep 18 20:21:33.642957 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:33.642867    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681c9102_7acf_4450_b2b7_2a65aac11518.slice/crio-1f28a6019808dc09c57f3b083a817edf535ac0a3136101aa0fbb2b74d276bec3 WatchSource:0}: Error finding container 1f28a6019808dc09c57f3b083a817edf535ac0a3136101aa0fbb2b74d276bec3: Status 404 returned error can't find the container with id 1f28a6019808dc09c57f3b083a817edf535ac0a3136101aa0fbb2b74d276bec3
Sep 18 20:21:33.645002 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:33.644980    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd1cf35_e51f_44ec_84fb_d0e22477bbe9.slice/crio-13513d7e81665f9f1ee60fbac44f90305849691822d3599afafde610307d7be6 WatchSource:0}: Error finding container 13513d7e81665f9f1ee60fbac44f90305849691822d3599afafde610307d7be6: Status 404 returned error can't find the container with id 13513d7e81665f9f1ee60fbac44f90305849691822d3599afafde610307d7be6
Sep 18 20:21:33.707548 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.707527    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mxhpq"
Sep 18 20:21:33.921479 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.921419    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-9bhvr" event={"ID":"681c9102-7acf-4450-b2b7-2a65aac11518","Type":"ContainerStarted","Data":"1f28a6019808dc09c57f3b083a817edf535ac0a3136101aa0fbb2b74d276bec3"}
Sep 18 20:21:33.923335 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.923290    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7548dd9977-tkk6p" event={"ID":"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9","Type":"ContainerStarted","Data":"fae850e56881caf2f3dc96cb69450d6524d66baf4f192b3a2e9cc440bfb5d1c4"}
Sep 18 20:21:33.923490 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.923339    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7548dd9977-tkk6p" event={"ID":"2cd1cf35-e51f-44ec-84fb-d0e22477bbe9","Type":"ContainerStarted","Data":"13513d7e81665f9f1ee60fbac44f90305849691822d3599afafde610307d7be6"}
Sep 18 20:21:33.923490 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.923422    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:33.925233 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.925184    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jlz2j" event={"ID":"f94eddb8-352a-4eaa-9301-26a28b73b7fe","Type":"ContainerStarted","Data":"72852493cbdac0354ebbce37d1bd87344b6fea3786d0ab66911a4d3e44fe0ac2"}
Sep 18 20:21:33.925377 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.925248    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jlz2j" event={"ID":"f94eddb8-352a-4eaa-9301-26a28b73b7fe","Type":"ContainerStarted","Data":"2c1fc65ffe686f96557e3a3701af0056b603e2efd2fed5271bcbe1c6a407d3ff"}
Sep 18 20:21:33.926735 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.926701    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf" event={"ID":"7b1188d9-c781-44ce-9fb8-4534f6f1e0e6","Type":"ContainerStarted","Data":"229b52045a7f161963dc645e71d1ce8364fd8fe69b30067a4892b63c620628c1"}
Sep 18 20:21:33.928122 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.928095    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j6v8w" event={"ID":"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf","Type":"ContainerStarted","Data":"e21440b834829ad3ff24410220eae028145b49160039300fc347a57f4b52af16"}
Sep 18 20:21:33.945163 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:33.945061    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7548dd9977-tkk6p" podStartSLOduration=1.9450419349999999 podStartE2EDuration="1.945041935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:33.944730336 +0000 UTC m=+57.438082936" watchObservedRunningTime="2026-09-18 20:21:33.945041935 +0000 UTC m=+57.438394510"
Sep 18 20:21:34.001233 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:34.001162    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mxhpq"]
Sep 18 20:21:34.004704 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:34.004677    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34aed437_eb0e_42d7_94fe_4829714cecc6.slice/crio-f054acccd76f642f2e5fbcca7a0ca52fcdae2eb8ff41fc950bb64c2bb20594ff WatchSource:0}: Error finding container f054acccd76f642f2e5fbcca7a0ca52fcdae2eb8ff41fc950bb64c2bb20594ff: Status 404 returned error can't find the container with id f054acccd76f642f2e5fbcca7a0ca52fcdae2eb8ff41fc950bb64c2bb20594ff
Sep 18 20:21:34.932275 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:34.932234    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mxhpq" event={"ID":"34aed437-eb0e-42d7-94fe-4829714cecc6","Type":"ContainerStarted","Data":"f054acccd76f642f2e5fbcca7a0ca52fcdae2eb8ff41fc950bb64c2bb20594ff"}
Sep 18 20:21:35.938289 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:35.938247    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jlz2j" event={"ID":"f94eddb8-352a-4eaa-9301-26a28b73b7fe","Type":"ContainerStarted","Data":"45345a6aac1f4fb7c0be73c2e9cd1a6a89937dcd577abf19ae10a36dcc976bf4"}
Sep 18 20:21:35.940180 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:35.940151    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf" event={"ID":"7b1188d9-c781-44ce-9fb8-4534f6f1e0e6","Type":"ContainerStarted","Data":"69f2def29d4a0f384dd95f2d2a4f2de5ec9cb18f457052dad70d4784ce3be986"}
Sep 18 20:21:35.940860 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:35.940810    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf"
Sep 18 20:21:35.946590 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:35.946565    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf"
Sep 18 20:21:35.958038 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:35.957990    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hzclf" podStartSLOduration=1.779151486 podStartE2EDuration="3.957976803s" totalImagesPullingTime="2.178825317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:35.957303489 +0000 UTC m=+59.450656072" watchObservedRunningTime="2026-09-18 20:21:35.957976803 +0000 UTC m=+59.451329381"
Sep 18 20:21:36.643336 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.643276    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:36.672768 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.672655    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"]
Sep 18 20:21:36.675366 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.675321    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.677709 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.677682    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 20:21:36.677833 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.677739    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-w9lff\""
Sep 18 20:21:36.678019 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.677956    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 20:21:36.678440 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.678404    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 20:21:36.678556 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.678499    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 20:21:36.678764 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.678717    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 20:21:36.685172 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.685149    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"]
Sep 18 20:21:36.745239 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.745206    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:36.811184 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.811151    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tms\" (UniqueName: \"kubernetes.io/projected/33a265ff-d032-46d3-832e-81d6b02d99a8-kube-api-access-l7tms\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.811376 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.811199    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33a265ff-d032-46d3-832e-81d6b02d99a8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.811376 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.811343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/33a265ff-d032-46d3-832e-81d6b02d99a8-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.811567 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.811375    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33a265ff-d032-46d3-832e-81d6b02d99a8-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.844398 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.844364    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:36.912473 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.912387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/33a265ff-d032-46d3-832e-81d6b02d99a8-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.912473 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.912441    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33a265ff-d032-46d3-832e-81d6b02d99a8-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.912678 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.912495    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tms\" (UniqueName: \"kubernetes.io/projected/33a265ff-d032-46d3-832e-81d6b02d99a8-kube-api-access-l7tms\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.912678 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.912520    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33a265ff-d032-46d3-832e-81d6b02d99a8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.913316 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.913291    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33a265ff-d032-46d3-832e-81d6b02d99a8-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.915475 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.915401    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33a265ff-d032-46d3-832e-81d6b02d99a8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.915573 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.915520    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/33a265ff-d032-46d3-832e-81d6b02d99a8-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.921909 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.921887    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tms\" (UniqueName: \"kubernetes.io/projected/33a265ff-d032-46d3-832e-81d6b02d99a8-kube-api-access-l7tms\") pod \"prometheus-operator-7bd5d56d6-6qxh7\" (UID: \"33a265ff-d032-46d3-832e-81d6b02d99a8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:36.946695 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.946666    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j6v8w" event={"ID":"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf","Type":"ContainerStarted","Data":"4d3344f19141788ba14e1386328f667b1ccf9009a1d3f6a4893f47e9cdde4ee4"}
Sep 18 20:21:36.947046 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.946724    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j6v8w" event={"ID":"d3a5a620-20ab-44e2-a5dd-89b7c7969dcf","Type":"ContainerStarted","Data":"cf5917b63082b1b1bd4aac151b9bac16fc321a6a5f66f4459c4202d6180d4503"}
Sep 18 20:21:36.947046 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.946832    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:36.972684 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.972630    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j6v8w" podStartSLOduration=2.577508572 podStartE2EDuration="4.972610533s" totalImagesPullingTime="2.395101961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:36.97141874 +0000 UTC m=+60.464771320" watchObservedRunningTime="2026-09-18 20:21:36.972610533 +0000 UTC m=+60.465963112"
Sep 18 20:21:36.988809 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:36.988788    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"
Sep 18 20:21:37.864050 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:37.864025    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5vk2"
Sep 18 20:21:37.952494 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:37.952438    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mxhpq" event={"ID":"34aed437-eb0e-42d7-94fe-4829714cecc6","Type":"ContainerStarted","Data":"48f3e84427280bde6b796bb2ff22d4fd7e8076c1b816b1c342d1bb0fa0307232"}
Sep 18 20:21:37.954950 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:37.954920    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jlz2j" event={"ID":"f94eddb8-352a-4eaa-9301-26a28b73b7fe","Type":"ContainerStarted","Data":"8ca433bf1b8d042bee382499a7ae1dc7170ddb1489d38a4ed40c91da0975de2c"}
Sep 18 20:21:37.968941 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:37.968882    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mxhpq" podStartSLOduration=1.206166551 podStartE2EDuration="4.968862812s" totalImagesPullingTime="3.762696261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:37.968409078 +0000 UTC m=+61.461761659" watchObservedRunningTime="2026-09-18 20:21:37.968862812 +0000 UTC m=+61.462215400"
Sep 18 20:21:37.983824 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:37.983797    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:37.988192 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:37.988148    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jlz2j" podStartSLOduration=2.083313012 podStartE2EDuration="5.988129245s" totalImagesPullingTime="3.904816233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:21:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:37.987524224 +0000 UTC m=+61.480876802" watchObservedRunningTime="2026-09-18 20:21:37.988129245 +0000 UTC m=+61.481481815"
Sep 18 20:21:38.043398 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:38.043319    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7"]
Sep 18 20:21:38.046904 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:38.046865    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a265ff_d032_46d3_832e_81d6b02d99a8.slice/crio-4513e69185d1c9f5b5c56a40017bfd47f891d1364954e1d3c9233aa3412d54f3 WatchSource:0}: Error finding container 4513e69185d1c9f5b5c56a40017bfd47f891d1364954e1d3c9233aa3412d54f3: Status 404 returned error can't find the container with id 4513e69185d1c9f5b5c56a40017bfd47f891d1364954e1d3c9233aa3412d54f3
Sep 18 20:21:38.052351 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:38.052328    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:38.960149 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:38.960109    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7" event={"ID":"33a265ff-d032-46d3-832e-81d6b02d99a8","Type":"ContainerStarted","Data":"4513e69185d1c9f5b5c56a40017bfd47f891d1364954e1d3c9233aa3412d54f3"}
Sep 18 20:21:39.652696 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:39.652590    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:39.966074 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:39.966032    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7" event={"ID":"33a265ff-d032-46d3-832e-81d6b02d99a8","Type":"ContainerStarted","Data":"9cf16390c4332f87a1a55c5274a0cce9df223952025f6eb350e9e9f254b6b405"}
Sep 18 20:21:39.966490 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:39.966099    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7" event={"ID":"33a265ff-d032-46d3-832e-81d6b02d99a8","Type":"ContainerStarted","Data":"fb5640a0987cfb6132dcd8645553295cdd68a6413e39bd8f297390e3a5418046"}
Sep 18 20:21:39.994355 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:39.994306    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6qxh7" podStartSLOduration=2.706504715 podStartE2EDuration="3.994289497s" totalImagesPullingTime="1.287784782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:39.993870749 +0000 UTC m=+63.487223328" watchObservedRunningTime="2026-09-18 20:21:39.994289497 +0000 UTC m=+63.487642075"
Sep 18 20:21:40.427912 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:40.427797    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:40.966799 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:40.966761    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:41.148685 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:41.148624    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:41.249561 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:41.249444    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:41.467282 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:41.467209    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:41.571421 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:41.571332    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:41.793371 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:41.793331    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:42.058316 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.058255    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:42.195235 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.195204    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"]
Sep 18 20:21:42.197703 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.197678    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.206850 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.206821    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 20:21:42.207158 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.207140    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 20:21:42.207303 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.207204    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-972wj\""
Sep 18 20:21:42.211840 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.210038    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"]
Sep 18 20:21:42.213773 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.213753    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.218607 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.218577    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qm52n\""
Sep 18 20:21:42.218607 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.218592    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 20:21:42.218765 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.218585    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 20:21:42.219282 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.219264    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 20:21:42.223551 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.223532    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"]
Sep 18 20:21:42.250873 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.250847    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"]
Sep 18 20:21:42.256904 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.256878    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0044db96-9a73-4280-adf5-26da2757eb37-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.257015 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.256921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a6ceca8-214e-43d1-8201-7932354a5ade-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.257015 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.256963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0044db96-9a73-4280-adf5-26da2757eb37-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.257124 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.257012    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.257124 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.257050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.257124 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.257108    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wt68\" (UniqueName: \"kubernetes.io/projected/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-api-access-9wt68\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.257273 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.257168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.257273 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.257232    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxrv6\" (UniqueName: \"kubernetes.io/projected/0044db96-9a73-4280-adf5-26da2757eb37-kube-api-access-lxrv6\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.257372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.257279    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0044db96-9a73-4280-adf5-26da2757eb37-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.257372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.257307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6ceca8-214e-43d1-8201-7932354a5ade-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.258218 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.258186    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:42.259831 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.259760    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xgrh8"]
Sep 18 20:21:42.262199 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.262179    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.265903 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.265879    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 20:21:42.266237 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.266220    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fg9vd\""
Sep 18 20:21:42.266306 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.266220    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 20:21:42.266420 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.266402    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 20:21:42.345231 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.345157    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:42.358417 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.358350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0044db96-9a73-4280-adf5-26da2757eb37-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.358417 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.358380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a6ceca8-214e-43d1-8201-7932354a5ade-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.358417 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.358401    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-tls\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.358705 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.358436    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0044db96-9a73-4280-adf5-26da2757eb37-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.358852 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.358815    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc94692c-fefc-48ff-a8a7-c7938555cd32-root\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.358926 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.358853    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a6ceca8-214e-43d1-8201-7932354a5ade-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.358926 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.358878    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.358926 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.358909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-wtmp\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.359078 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.358937    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-accelerators-collector-config\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.359078 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.358991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.359078 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359065    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.359268 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359183    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wt68\" (UniqueName: \"kubernetes.io/projected/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-api-access-9wt68\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.359268 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359205    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0044db96-9a73-4280-adf5-26da2757eb37-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.359431 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359262    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-textfile\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.359431 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359296    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.359431 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359333    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc94692c-fefc-48ff-a8a7-c7938555cd32-sys\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.359431 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359388    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxrv6\" (UniqueName: \"kubernetes.io/projected/0044db96-9a73-4280-adf5-26da2757eb37-kube-api-access-lxrv6\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.359684 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359437    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0044db96-9a73-4280-adf5-26da2757eb37-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.359684 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359486    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6ceca8-214e-43d1-8201-7932354a5ade-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.359684 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359510    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk7fq\" (UniqueName: \"kubernetes.io/projected/dc94692c-fefc-48ff-a8a7-c7938555cd32-kube-api-access-pk7fq\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.359684 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.359543    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc94692c-fefc-48ff-a8a7-c7938555cd32-metrics-client-ca\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.359684 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:42.359609    2557 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 18 20:21:42.359684 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:42.359673    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0044db96-9a73-4280-adf5-26da2757eb37-openshift-state-metrics-tls podName:0044db96-9a73-4280-adf5-26da2757eb37 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:42.859653344 +0000 UTC m=+66.353005900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0044db96-9a73-4280-adf5-26da2757eb37-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-gchwm" (UID: "0044db96-9a73-4280-adf5-26da2757eb37") : secret "openshift-state-metrics-tls" not found
Sep 18 20:21:42.360096 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.360067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.360242 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.360220    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6ceca8-214e-43d1-8201-7932354a5ade-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.361476 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.361437    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.361585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.361539    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0044db96-9a73-4280-adf5-26da2757eb37-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.361807 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.361791    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.368142 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.368113    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxrv6\" (UniqueName: \"kubernetes.io/projected/0044db96-9a73-4280-adf5-26da2757eb37-kube-api-access-lxrv6\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.368714 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.368694    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wt68\" (UniqueName: \"kubernetes.io/projected/6a6ceca8-214e-43d1-8201-7932354a5ade-kube-api-access-9wt68\") pod \"kube-state-metrics-84c885658c-7k4q5\" (UID: \"6a6ceca8-214e-43d1-8201-7932354a5ade\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.460759 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.460731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-tls\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.460909 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.460786    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc94692c-fefc-48ff-a8a7-c7938555cd32-root\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.460909 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.460818    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-wtmp\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461032 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.460932    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-wtmp\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461032 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.460948    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc94692c-fefc-48ff-a8a7-c7938555cd32-root\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461032 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.460994    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-accelerators-collector-config\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461188 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.461039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461188 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.461059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-textfile\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461188 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.461084    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc94692c-fefc-48ff-a8a7-c7938555cd32-sys\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461306 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.461171    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk7fq\" (UniqueName: \"kubernetes.io/projected/dc94692c-fefc-48ff-a8a7-c7938555cd32-kube-api-access-pk7fq\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461306 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.461220    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc94692c-fefc-48ff-a8a7-c7938555cd32-metrics-client-ca\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461306 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.461222    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc94692c-fefc-48ff-a8a7-c7938555cd32-sys\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461444 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.461388    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-textfile\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461715 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.461665    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-accelerators-collector-config\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.461830 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.461754    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc94692c-fefc-48ff-a8a7-c7938555cd32-metrics-client-ca\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.463611 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.463565    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.463611 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.463563    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc94692c-fefc-48ff-a8a7-c7938555cd32-node-exporter-tls\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.470052 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.470029    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk7fq\" (UniqueName: \"kubernetes.io/projected/dc94692c-fefc-48ff-a8a7-c7938555cd32-kube-api-access-pk7fq\") pod \"node-exporter-xgrh8\" (UID: \"dc94692c-fefc-48ff-a8a7-c7938555cd32\") " pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.522324 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.522297    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:42.523186 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.523155    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"
Sep 18 20:21:42.573283 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.573254    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xgrh8"
Sep 18 20:21:42.588036 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:42.588010    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc94692c_fefc_48ff_a8a7_c7938555cd32.slice/crio-6c34ead1d43ab7e76b34f4769572a2e2888ed0a6683627647e658c0e21d0c27f WatchSource:0}: Error finding container 6c34ead1d43ab7e76b34f4769572a2e2888ed0a6683627647e658c0e21d0c27f: Status 404 returned error can't find the container with id 6c34ead1d43ab7e76b34f4769572a2e2888ed0a6683627647e658c0e21d0c27f
Sep 18 20:21:42.619340 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.619280    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:42.720696 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.720660    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:42.812150 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.812121    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-7k4q5"]
Sep 18 20:21:42.815276 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:42.815247    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6ceca8_214e_43d1_8201_7932354a5ade.slice/crio-9d9cc9ab40639de58c990089a103f18a482078fbb8190dbb5576b359f1a7f57c WatchSource:0}: Error finding container 9d9cc9ab40639de58c990089a103f18a482078fbb8190dbb5576b359f1a7f57c: Status 404 returned error can't find the container with id 9d9cc9ab40639de58c990089a103f18a482078fbb8190dbb5576b359f1a7f57c
Sep 18 20:21:42.865502 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.865473    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0044db96-9a73-4280-adf5-26da2757eb37-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.867910 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.867889    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0044db96-9a73-4280-adf5-26da2757eb37-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-gchwm\" (UID: \"0044db96-9a73-4280-adf5-26da2757eb37\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:42.979530 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.979479    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5" event={"ID":"6a6ceca8-214e-43d1-8201-7932354a5ade","Type":"ContainerStarted","Data":"9d9cc9ab40639de58c990089a103f18a482078fbb8190dbb5576b359f1a7f57c"}
Sep 18 20:21:42.981023 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.980995    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xgrh8" event={"ID":"dc94692c-fefc-48ff-a8a7-c7938555cd32","Type":"ContainerStarted","Data":"6c34ead1d43ab7e76b34f4769572a2e2888ed0a6683627647e658c0e21d0c27f"}
Sep 18 20:21:42.990426 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:42.990397    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:43.088808 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.088775    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:43.093957 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.093848    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:21:43.097105 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.097078    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.099482 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.099433    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 20:21:43.099623 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.099601    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 20:21:43.099696 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.099670    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 20:21:43.099920 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.099902    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 20:21:43.100267 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.100118    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 20:21:43.100267 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.100202    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 20:21:43.100650 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.100525    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m85gr\""
Sep 18 20:21:43.100650 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.100549    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 20:21:43.100814 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.100798    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 20:21:43.106807 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.106788    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 20:21:43.108687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.108668    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"
Sep 18 20:21:43.117317 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.117288    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:21:43.168325 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168283    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168643 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168356    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168643 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168393    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168643 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168429    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168643 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168478    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168643 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168559    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-web-config\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168643 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168605    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-config-out\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168978 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168650    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168978 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168699    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168978 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168749    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-config-volume\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168978 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168773    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168978 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168822    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwt7\" (UniqueName: \"kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-kube-api-access-vhwt7\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.168978 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.168846    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270522 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270227    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270522 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270282    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-web-config\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270522 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270318    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-config-out\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270522 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270522 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270386    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270522 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:43.270428    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-trusted-ca-bundle podName:d81edf6f-0611-4809-b514-323afaabfbb7 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:43.77040409 +0000 UTC m=+67.263756650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7") : configmap references non-existent config key: ca-bundle.crt
Sep 18 20:21:43.270522 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-config-volume\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270989 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:43.270533    2557 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 18 20:21:43.270989 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270989 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:21:43.270591    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-main-tls podName:d81edf6f-0611-4809-b514-323afaabfbb7 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:43.770573151 +0000 UTC m=+67.263925714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7") : secret "alertmanager-main-tls" not found
Sep 18 20:21:43.270989 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270624    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwt7\" (UniqueName: \"kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-kube-api-access-vhwt7\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270989 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270989 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270697    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270989 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270733    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270989 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270767    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.270989 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.270800    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.271431 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.271195    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.271976 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.271909    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.273382 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.273337    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.274737 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.274690    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.275488 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.275144    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-config-out\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.275488 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.275439    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.275642 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.275518    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.276142 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.276107    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-config-volume\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.277913 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.277875    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-web-config\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.278260 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.278240    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.280951 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.280911    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwt7\" (UniqueName: \"kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-kube-api-access-vhwt7\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.401612 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.401495    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm"]
Sep 18 20:21:43.523445 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.523321    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:43.619513 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:43.619472    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0044db96_9a73_4280_adf5_26da2757eb37.slice/crio-a26c860dbe7a15678eb21a3a57c4e510752f1509d4c5a2ac7f7d7ba15a90db7e WatchSource:0}: Error finding container a26c860dbe7a15678eb21a3a57c4e510752f1509d4c5a2ac7f7d7ba15a90db7e: Status 404 returned error can't find the container with id a26c860dbe7a15678eb21a3a57c4e510752f1509d4c5a2ac7f7d7ba15a90db7e
Sep 18 20:21:43.783771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.783697    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.783771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.783753    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.784929 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.784893    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.786014 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.785991    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:43.986945 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.986885    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm" event={"ID":"0044db96-9a73-4280-adf5-26da2757eb37","Type":"ContainerStarted","Data":"bb0fb006eef3db707561a037084971ceb9f1283d5f0d0f1b3329b1b896bdddee"}
Sep 18 20:21:43.987126 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.986962    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm" event={"ID":"0044db96-9a73-4280-adf5-26da2757eb37","Type":"ContainerStarted","Data":"a26c860dbe7a15678eb21a3a57c4e510752f1509d4c5a2ac7f7d7ba15a90db7e"}
Sep 18 20:21:43.988777 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:43.988747    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xgrh8" event={"ID":"dc94692c-fefc-48ff-a8a7-c7938555cd32","Type":"ContainerStarted","Data":"28696a9ddbcffa2b8f6b2a0dd26b2512db4e025d325744439ed9cb423ca2342f"}
Sep 18 20:21:44.009324 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.009301    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:44.490107 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.490070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:44.490548 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.490186    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:44.492585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.492563    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 20:21:44.492999 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.492979    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 20:21:44.502860 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.502836    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e506cbe-bfeb-4614-b6c7-91b349785053-metrics-certs\") pod \"network-metrics-daemon-7kdrq\" (UID: \"4e506cbe-bfeb-4614-b6c7-91b349785053\") " pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:44.502947 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.502859    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62a9d73d-1877-48b9-94e2-5a76311b7453-original-pull-secret\") pod \"global-pull-secret-syncer-l6x2p\" (UID: \"62a9d73d-1877-48b9-94e2-5a76311b7453\") " pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:44.591438 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.591399    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqv\" (UniqueName: \"kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv\") pod \"network-check-target-bm5dj\" (UID: \"edb74921-50a4-402a-b2cc-138778d94d25\") " pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:44.594108 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.594087    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 20:21:44.604500 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.604480    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 20:21:44.615531 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.615479    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmqv\" (UniqueName: \"kubernetes.io/projected/edb74921-50a4-402a-b2cc-138778d94d25-kube-api-access-wvmqv\") pod \"network-check-target-bm5dj\" (UID: \"edb74921-50a4-402a-b2cc-138778d94d25\") " pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:44.627339 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.627316    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6x2p"
Sep 18 20:21:44.634809 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.634788    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnbxc\""
Sep 18 20:21:44.643500 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.643477    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kdrq"
Sep 18 20:21:44.645754 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.645737    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r5tl8\""
Sep 18 20:21:44.646781 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.646761    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:21:44.994648 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.994611    2557 generic.go:356] "Generic (PLEG): container finished" podID="dc94692c-fefc-48ff-a8a7-c7938555cd32" containerID="28696a9ddbcffa2b8f6b2a0dd26b2512db4e025d325744439ed9cb423ca2342f" exitCode=0
Sep 18 20:21:44.994836 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:44.994669    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xgrh8" event={"ID":"dc94692c-fefc-48ff-a8a7-c7938555cd32","Type":"ContainerDied","Data":"28696a9ddbcffa2b8f6b2a0dd26b2512db4e025d325744439ed9cb423ca2342f"}
Sep 18 20:21:45.204937 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:45.204875    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:45.303781 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:45.303707    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:45.646789 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:45.646711    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:46.561050 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.561012    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:46.636552 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.636518    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-554d6d74df-lmtzk"]
Sep 18 20:21:46.638656 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.638639    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.640643 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.640600    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 20:21:46.640773 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.640679    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jc5bf\""
Sep 18 20:21:46.640773 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.640698    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 20:21:46.645107 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.644789    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fu01m60f8920q\""
Sep 18 20:21:46.645107 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.644869    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 20:21:46.645107 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.644977    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:46.645317 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.645131    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 20:21:46.653048 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.653029    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-554d6d74df-lmtzk"]
Sep 18 20:21:46.712957 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.712922    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-secret-metrics-server-client-certs\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.713119 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.712969    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-audit-log\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.713119 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.712999    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-client-ca-bundle\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.713119 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.713039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.713266 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.713176    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nln8\" (UniqueName: \"kubernetes.io/projected/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-kube-api-access-2nln8\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.713266 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.713218    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-metrics-server-audit-profiles\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.713347 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.713267    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-secret-metrics-server-tls\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.814307 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.814234    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-secret-metrics-server-client-certs\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.814307 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.814272    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-audit-log\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.814307 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.814291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-client-ca-bundle\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.814666 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.814318    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.814666 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.814367    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nln8\" (UniqueName: \"kubernetes.io/projected/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-kube-api-access-2nln8\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.814666 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.814395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-metrics-server-audit-profiles\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.814666 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.814431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-secret-metrics-server-tls\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.814892 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.814710    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-audit-log\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.815141 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.815108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.815493 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.815477    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-metrics-server-audit-profiles\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.817203 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.817167    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-secret-metrics-server-client-certs\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.817302 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.817283    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-secret-metrics-server-tls\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.817418 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.817393    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-client-ca-bundle\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.819337 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.819292    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:46.820908 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.820882    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"]
Sep 18 20:21:46.823884 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.823865    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"
Sep 18 20:21:46.826122 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.826101    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pkcp5\""
Sep 18 20:21:46.826219 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.826144    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 20:21:46.829437 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.829412    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nln8\" (UniqueName: \"kubernetes.io/projected/3859bc03-5295-4b0f-90f7-f0d1a8b2df5f-kube-api-access-2nln8\") pod \"metrics-server-554d6d74df-lmtzk\" (UID: \"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f\") " pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.832343 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.832321    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"]
Sep 18 20:21:46.915862 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.915830    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a488e4e5-0b3b-4e74-b4c4-38d29c1243f4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-qzxjb\" (UID: \"a488e4e5-0b3b-4e74-b4c4-38d29c1243f4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"
Sep 18 20:21:46.916016 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.915876    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz9cb\" (UniqueName: \"kubernetes.io/projected/a488e4e5-0b3b-4e74-b4c4-38d29c1243f4-kube-api-access-lz9cb\") pod \"monitoring-plugin-5984459ff8-qzxjb\" (UID: \"a488e4e5-0b3b-4e74-b4c4-38d29c1243f4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"
Sep 18 20:21:46.954573 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.954543    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:21:46.957609 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.957587    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j6v8w"
Sep 18 20:21:46.958800 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:46.958771    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:47.017038 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:47.016798    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a488e4e5-0b3b-4e74-b4c4-38d29c1243f4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-qzxjb\" (UID: \"a488e4e5-0b3b-4e74-b4c4-38d29c1243f4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"
Sep 18 20:21:47.017038 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:47.016837    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz9cb\" (UniqueName: \"kubernetes.io/projected/a488e4e5-0b3b-4e74-b4c4-38d29c1243f4-kube-api-access-lz9cb\") pod \"monitoring-plugin-5984459ff8-qzxjb\" (UID: \"a488e4e5-0b3b-4e74-b4c4-38d29c1243f4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"
Sep 18 20:21:47.019482 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:47.019437    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a488e4e5-0b3b-4e74-b4c4-38d29c1243f4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-qzxjb\" (UID: \"a488e4e5-0b3b-4e74-b4c4-38d29c1243f4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"
Sep 18 20:21:47.031192 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:47.031144    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz9cb\" (UniqueName: \"kubernetes.io/projected/a488e4e5-0b3b-4e74-b4c4-38d29c1243f4-kube-api-access-lz9cb\") pod \"monitoring-plugin-5984459ff8-qzxjb\" (UID: \"a488e4e5-0b3b-4e74-b4c4-38d29c1243f4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"
Sep 18 20:21:47.082204 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:47.082141    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:47.146825 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:47.146794    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"
Sep 18 20:21:48.334567 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:48.334531    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:48.454412 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:48.454376    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:48.711799 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:48.711762    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:48.874482 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:48.874422    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:50.580718 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.580683    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-584fc9568d-gp2fh"]
Sep 18 20:21:50.583047 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.583028    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.589168 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.589143    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bf7wk\""
Sep 18 20:21:50.589479 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.589381    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 20:21:50.589684 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.589659    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 20:21:50.589792 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.589681    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 20:21:50.589792 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.589767    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 20:21:50.591027 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.590964    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 20:21:50.594972 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.594946    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 20:21:50.609902 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.609879    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-584fc9568d-gp2fh"]
Sep 18 20:21:50.651562 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.651541    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-service-ca\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.651682 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.651612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-config\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.651682 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.651660    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-trusted-ca-bundle\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.651797 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.651774    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-serving-cert\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.651854 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.651813    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-oauth-serving-cert\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.651854 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.651847    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-oauth-config\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.651939 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.651919    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jlw2\" (UniqueName: \"kubernetes.io/projected/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-kube-api-access-8jlw2\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.753226 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.753194    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-config\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.753395 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.753236    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-trusted-ca-bundle\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.753483 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.753395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-serving-cert\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.753483 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.753438    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-oauth-serving-cert\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.753605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.753489    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-oauth-config\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.753605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.753552    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlw2\" (UniqueName: \"kubernetes.io/projected/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-kube-api-access-8jlw2\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.753605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.753582    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-service-ca\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.754082 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.754039    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-config\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.754177 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.754115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-trusted-ca-bundle\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.754295 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.754273    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-service-ca\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.754539 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.754497    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-oauth-serving-cert\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.756580 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.756556    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-serving-cert\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.756694 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.756583    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-oauth-config\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.763779 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.763756    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jlw2\" (UniqueName: \"kubernetes.io/projected/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-kube-api-access-8jlw2\") pod \"console-584fc9568d-gp2fh\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") " pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:50.894374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:50.894295    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:21:51.395079 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:51.395038    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-554d6d74df-lmtzk"]
Sep 18 20:21:51.400853 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:51.400817    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3859bc03_5295_4b0f_90f7_f0d1a8b2df5f.slice/crio-6695b445329e4d5f62594066c584682fb5dd4349e288665508e6fc176f10efa0 WatchSource:0}: Error finding container 6695b445329e4d5f62594066c584682fb5dd4349e288665508e6fc176f10efa0: Status 404 returned error can't find the container with id 6695b445329e4d5f62594066c584682fb5dd4349e288665508e6fc176f10efa0
Sep 18 20:21:51.436567 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:51.436527    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l6x2p"]
Sep 18 20:21:51.440176 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:51.440151    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a9d73d_1877_48b9_94e2_5a76311b7453.slice/crio-da022a3ff27f6af1c90e5c5acdc4ac03284e4a265adb150f2a282e56b622b1f8 WatchSource:0}: Error finding container da022a3ff27f6af1c90e5c5acdc4ac03284e4a265adb150f2a282e56b622b1f8: Status 404 returned error can't find the container with id da022a3ff27f6af1c90e5c5acdc4ac03284e4a265adb150f2a282e56b622b1f8
Sep 18 20:21:51.452304 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:51.452274    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-584fc9568d-gp2fh"]
Sep 18 20:21:51.463533 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:51.463509    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bm5dj"]
Sep 18 20:21:51.469658 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:51.469636    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7kdrq"]
Sep 18 20:21:51.473131 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:51.473113    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:21:51.479266 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:51.479194    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81edf6f_0611_4809_b514_323afaabfbb7.slice/crio-33f3f067257e9884618e213032b6bad4c5c2032e7ca5750b7b050306beb2754d WatchSource:0}: Error finding container 33f3f067257e9884618e213032b6bad4c5c2032e7ca5750b7b050306beb2754d: Status 404 returned error can't find the container with id 33f3f067257e9884618e213032b6bad4c5c2032e7ca5750b7b050306beb2754d
Sep 18 20:21:51.514546 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:51.514514    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda488e4e5_0b3b_4e74_b4c4_38d29c1243f4.slice/crio-7203c2819e8e7f71dfa4de40a6a457b8a339c9c766e02b3a096b1c64ecdd7051 WatchSource:0}: Error finding container 7203c2819e8e7f71dfa4de40a6a457b8a339c9c766e02b3a096b1c64ecdd7051: Status 404 returned error can't find the container with id 7203c2819e8e7f71dfa4de40a6a457b8a339c9c766e02b3a096b1c64ecdd7051
Sep 18 20:21:51.528838 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:51.528811    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"]
Sep 18 20:21:52.033214 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.033165    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-9bhvr" event={"ID":"681c9102-7acf-4450-b2b7-2a65aac11518","Type":"ContainerStarted","Data":"2346b45882da66f5a3901139e3130254f05057234700c97c6ce67210c5272cb6"}
Sep 18 20:21:52.034312 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.034281    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-9bhvr"
Sep 18 20:21:52.040758 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.040203    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7kdrq" event={"ID":"4e506cbe-bfeb-4614-b6c7-91b349785053","Type":"ContainerStarted","Data":"67f4e9a5200924eb3960b0b96ba03a402345f14b3f530a1e724d3c8de48db1ef"}
Sep 18 20:21:52.044184 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.044091    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerStarted","Data":"33f3f067257e9884618e213032b6bad4c5c2032e7ca5750b7b050306beb2754d"}
Sep 18 20:21:52.046095 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.046031    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fc9568d-gp2fh" event={"ID":"2a9016ff-55c9-42fe-ad91-3d0edab72fd5","Type":"ContainerStarted","Data":"89fbdc838f031576e4091b11505ce1d29bf134f72048c2b507b1b9c68bd3e73d"}
Sep 18 20:21:52.047209 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.047169    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-9bhvr"
Sep 18 20:21:52.057750 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.056977    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5" event={"ID":"6a6ceca8-214e-43d1-8201-7932354a5ade","Type":"ContainerStarted","Data":"5f922b9649232da472d3640b8fa00428410ac0f3ec434c5bd020f19cec8e5b25"}
Sep 18 20:21:52.057750 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.057033    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5" event={"ID":"6a6ceca8-214e-43d1-8201-7932354a5ade","Type":"ContainerStarted","Data":"2c98c5c2f717457603ec988256e40f82f6df51c755edc3539c9e40c11fa3983e"}
Sep 18 20:21:52.057750 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.057065    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5" event={"ID":"6a6ceca8-214e-43d1-8201-7932354a5ade","Type":"ContainerStarted","Data":"a19ae219265c3db4d3f50838ee7c0d991b34491ee804f7ecd06b097acf14edd6"}
Sep 18 20:21:52.066586 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.066540    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm" event={"ID":"0044db96-9a73-4280-adf5-26da2757eb37","Type":"ContainerStarted","Data":"38cdd7c51dce5e38842ad69fe7d0ea0a2e889ddd38196b69ad4c27e8f58d6c3e"}
Sep 18 20:21:52.069143 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.069065    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l6x2p" event={"ID":"62a9d73d-1877-48b9-94e2-5a76311b7453","Type":"ContainerStarted","Data":"da022a3ff27f6af1c90e5c5acdc4ac03284e4a265adb150f2a282e56b622b1f8"}
Sep 18 20:21:52.072917 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.072854    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk" event={"ID":"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f","Type":"ContainerStarted","Data":"6695b445329e4d5f62594066c584682fb5dd4349e288665508e6fc176f10efa0"}
Sep 18 20:21:52.083037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.082611    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xgrh8" event={"ID":"dc94692c-fefc-48ff-a8a7-c7938555cd32","Type":"ContainerStarted","Data":"a19baffb7dc080336c89289477a8cc48a3aa2760ae0488246eba179590aa94cf"}
Sep 18 20:21:52.083037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.082660    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xgrh8" event={"ID":"dc94692c-fefc-48ff-a8a7-c7938555cd32","Type":"ContainerStarted","Data":"af180c738cc350ab1678de1b7ceb0cf3f938ae4174c0f8a63e3e340a832b8c01"}
Sep 18 20:21:52.085559 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.085529    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb" event={"ID":"a488e4e5-0b3b-4e74-b4c4-38d29c1243f4","Type":"ContainerStarted","Data":"7203c2819e8e7f71dfa4de40a6a457b8a339c9c766e02b3a096b1c64ecdd7051"}
Sep 18 20:21:52.094774 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.094714    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bm5dj" event={"ID":"edb74921-50a4-402a-b2cc-138778d94d25","Type":"ContainerStarted","Data":"971b23dab95b33cf1896a3d140f5cc7de42c63726e3ac6d13c45abea485ab2ba"}
Sep 18 20:21:52.116712 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.115221    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-7k4q5" podStartSLOduration=1.8504556920000002 podStartE2EDuration="10.115203599s" totalImagesPullingTime="8.264747907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:52.114613788 +0000 UTC m=+75.607966365" watchObservedRunningTime="2026-09-18 20:21:52.115203599 +0000 UTC m=+75.608556175"
Sep 18 20:21:52.116712 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.115529    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-9bhvr" podStartSLOduration=2.582796665 podStartE2EDuration="20.115519675s" totalImagesPullingTime="17.53272301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:52.070617476 +0000 UTC m=+75.563970054" watchObservedRunningTime="2026-09-18 20:21:52.115519675 +0000 UTC m=+75.608872253"
Sep 18 20:21:52.501704 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.501010    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xgrh8" podStartSLOduration=9.362290537 podStartE2EDuration="10.500984209s" totalImagesPullingTime="1.064908692s" totalInitContainerRuntime="73.78498ms" isStatefulPod=true podCreationTimestamp="2026-09-18 20:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:52.234763236 +0000 UTC m=+75.728115816" watchObservedRunningTime="2026-09-18 20:21:52.500984209 +0000 UTC m=+75.994336788"
Sep 18 20:21:52.501704 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.501629    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-584fc9568d-gp2fh"]
Sep 18 20:21:52.517079 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.516983    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:52.541228 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.539748    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86649cfc57-2zdzh"]
Sep 18 20:21:52.548207 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.548177    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.556180 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.555966    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86649cfc57-2zdzh"]
Sep 18 20:21:52.611363 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.611226    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:52.697250 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.696834    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-oauth-serving-cert\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.697250 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.696882    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-console-config\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.697250 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.696919    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4wl5\" (UniqueName: \"kubernetes.io/projected/a66eeb85-9624-4396-9c89-341d50626bc6-kube-api-access-h4wl5\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.697250 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.696959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-oauth-config\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.697250 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.697001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-serving-cert\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.697250 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.697057    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-trusted-ca-bundle\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.697250 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.697085    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-service-ca\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.802291 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.797928    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-oauth-serving-cert\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.802291 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.797979    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-console-config\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.802291 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.798012    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wl5\" (UniqueName: \"kubernetes.io/projected/a66eeb85-9624-4396-9c89-341d50626bc6-kube-api-access-h4wl5\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.802291 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.798053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-oauth-config\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.802291 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.798097    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-serving-cert\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.802291 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.798153    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-trusted-ca-bundle\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.802291 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.798180    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-service-ca\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.802291 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.799394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-oauth-serving-cert\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.808683 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.807328    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-trusted-ca-bundle\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.809270 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.809216    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-service-ca\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.809968 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.809920    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-console-config\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.847588 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.847530    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-serving-cert\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.847588 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.847557    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-oauth-config\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.858872 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.858811    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wl5\" (UniqueName: \"kubernetes.io/projected/a66eeb85-9624-4396-9c89-341d50626bc6-kube-api-access-h4wl5\") pod \"console-86649cfc57-2zdzh\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") " pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:52.868396 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:52.868126    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:21:54.940039 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:54.939787    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7548dd9977-tkk6p"
Sep 18 20:21:55.135640 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:55.135602    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:55.240854 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:55.240779    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:55.339973 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:55.339927    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:56.840882 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:56.840604    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86649cfc57-2zdzh"]
Sep 18 20:21:57.663829 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:57.663790    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:57.764215 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:57.764135    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:58.847018 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:21:58.846985    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66eeb85_9624_4396_9c89_341d50626bc6.slice/crio-a09a5f7824d450bdca21fcc8f68059229dc334b83d0ba6d0f8386e4a93e6a02d WatchSource:0}: Error finding container a09a5f7824d450bdca21fcc8f68059229dc334b83d0ba6d0f8386e4a93e6a02d: Status 404 returned error can't find the container with id a09a5f7824d450bdca21fcc8f68059229dc334b83d0ba6d0f8386e4a93e6a02d
Sep 18 20:21:59.145742 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:21:59.145652    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86649cfc57-2zdzh" event={"ID":"a66eeb85-9624-4396-9c89-341d50626bc6","Type":"ContainerStarted","Data":"a09a5f7824d450bdca21fcc8f68059229dc334b83d0ba6d0f8386e4a93e6a02d"}
Sep 18 20:22:00.596328 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:00.596292    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:00.696845 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:00.696779    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:01.687004 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:01.686966    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:01.777063 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:01.776928    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:05.172741 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.172705    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7kdrq" event={"ID":"4e506cbe-bfeb-4614-b6c7-91b349785053","Type":"ContainerStarted","Data":"aa55171cd5ac972d07b3ba3b888c25191b1dae6a14a226f336423592c48705aa"}
Sep 18 20:22:05.173236 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.172774    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7kdrq" event={"ID":"4e506cbe-bfeb-4614-b6c7-91b349785053","Type":"ContainerStarted","Data":"f092d059c4ab446eeaa9b4361af6bfa66323b9253fd178d41a60657accfba9e9"}
Sep 18 20:22:05.174175 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.174148    2557 generic.go:356] "Generic (PLEG): container finished" podID="d81edf6f-0611-4809-b514-323afaabfbb7" containerID="4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c" exitCode=0
Sep 18 20:22:05.174276 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.174221    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerDied","Data":"4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c"}
Sep 18 20:22:05.175908 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.175844    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l6x2p" event={"ID":"62a9d73d-1877-48b9-94e2-5a76311b7453","Type":"ContainerStarted","Data":"4b38e6c1b8e38f2176071d96ed1b2f47f6baf49cb7504c412f30324ef9b06b25"}
Sep 18 20:22:05.177408 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.177383    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk" event={"ID":"3859bc03-5295-4b0f-90f7-f0d1a8b2df5f","Type":"ContainerStarted","Data":"772e1cd525cf234fb8920b6b50e11b6012e1c22131aa51fb95e58e325923ef58"}
Sep 18 20:22:05.178985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.178962    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fc9568d-gp2fh" event={"ID":"2a9016ff-55c9-42fe-ad91-3d0edab72fd5","Type":"ContainerStarted","Data":"cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf"}
Sep 18 20:22:05.181267 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.181240    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm" event={"ID":"0044db96-9a73-4280-adf5-26da2757eb37","Type":"ContainerStarted","Data":"a831c31958a59c9ff9ede0fbd236b9a43fc775bb1d237df730840706bdef8de2"}
Sep 18 20:22:05.182945 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.182921    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86649cfc57-2zdzh" event={"ID":"a66eeb85-9624-4396-9c89-341d50626bc6","Type":"ContainerStarted","Data":"bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b"}
Sep 18 20:22:05.184441 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.184408    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb" event={"ID":"a488e4e5-0b3b-4e74-b4c4-38d29c1243f4","Type":"ContainerStarted","Data":"397d0055f62116d56bad126c5b56e7fa6993a194950e49e61e2d2209f02bc92a"}
Sep 18 20:22:05.184597 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.184576    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"
Sep 18 20:22:05.186101 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.186083    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bm5dj" event={"ID":"edb74921-50a4-402a-b2cc-138778d94d25","Type":"ContainerStarted","Data":"49adb7c84a435d60efe1bfbbddca2d656d46d474239291076fd2f73c716327d6"}
Sep 18 20:22:05.186228 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.186210    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:22:05.189541 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.189502    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7kdrq" podStartSLOduration=74.471385406 podStartE2EDuration="1m27.189489549s" totalImagesPullingTime="12.718104143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:05.187876464 +0000 UTC m=+88.681229043" watchObservedRunningTime="2026-09-18 20:22:05.189489549 +0000 UTC m=+88.682842120"
Sep 18 20:22:05.189934 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.189914    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb"
Sep 18 20:22:05.205743 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.205668    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-gchwm" podStartSLOduration=11.547829166 podStartE2EDuration="23.205656647s" totalImagesPullingTime="11.657827481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:05.204150046 +0000 UTC m=+88.697502624" watchObservedRunningTime="2026-09-18 20:22:05.205656647 +0000 UTC m=+88.699009221"
Sep 18 20:22:05.219883 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.219813    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l6x2p" podStartSLOduration=74.424933912 podStartE2EDuration="1m27.219802327s" totalImagesPullingTime="12.794868415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:05.218875975 +0000 UTC m=+88.712228553" watchObservedRunningTime="2026-09-18 20:22:05.219802327 +0000 UTC m=+88.713154904"
Sep 18 20:22:05.240336 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.240298    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk" podStartSLOduration=6.417688312 podStartE2EDuration="19.2402868s" totalImagesPullingTime="12.822598488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:05.239151941 +0000 UTC m=+88.732504546" watchObservedRunningTime="2026-09-18 20:22:05.2402868 +0000 UTC m=+88.733639379"
Sep 18 20:22:05.256489 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.256437    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-qzxjb" podStartSLOduration=8.688386056 podStartE2EDuration="19.256426566s" totalImagesPullingTime="10.56804051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:05.255719129 +0000 UTC m=+88.749071712" watchObservedRunningTime="2026-09-18 20:22:05.256426566 +0000 UTC m=+88.749779181"
Sep 18 20:22:05.280021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.279987    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-584fc9568d-gp2fh" podStartSLOduration=2.074367778 podStartE2EDuration="15.279973526s" totalImagesPullingTime="13.205605748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:05.279399781 +0000 UTC m=+88.772752372" watchObservedRunningTime="2026-09-18 20:22:05.279973526 +0000 UTC m=+88.773326103"
Sep 18 20:22:05.284302 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.284284    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:05.318356 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.318315    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bm5dj" podStartSLOduration=74.559301179 podStartE2EDuration="1m27.318303903s" totalImagesPullingTime="12.759002724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:05.317884325 +0000 UTC m=+88.811236903" watchObservedRunningTime="2026-09-18 20:22:05.318303903 +0000 UTC m=+88.811656481"
Sep 18 20:22:05.318629 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.318606    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86649cfc57-2zdzh" podStartSLOduration=7.358712137 podStartE2EDuration="13.318598196s" totalImagesPullingTime="5.959886059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:05.301745701 +0000 UTC m=+88.795098273" watchObservedRunningTime="2026-09-18 20:22:05.318598196 +0000 UTC m=+88.811950773"
Sep 18 20:22:05.361622 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.361572    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:05.428508 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:05.428474    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:06.954955 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:06.954919    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:22:06.955387 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:06.955002    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:22:07.198054 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:07.198025    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerStarted","Data":"7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a"}
Sep 18 20:22:08.205412 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:08.205374    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerStarted","Data":"e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2"}
Sep 18 20:22:08.205855 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:08.205478    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerStarted","Data":"23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516"}
Sep 18 20:22:08.205855 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:08.205524    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerStarted","Data":"d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423"}
Sep 18 20:22:08.205855 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:08.205553    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerStarted","Data":"912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45"}
Sep 18 20:22:09.212437 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:09.212401    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerStarted","Data":"725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e"}
Sep 18 20:22:09.248777 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:09.248732    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=11.70681708 podStartE2EDuration="26.248716131s" totalImagesPullingTime="14.528627461s" totalInitContainerRuntime="13.27159ms" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:09.246037813 +0000 UTC m=+92.739390392" watchObservedRunningTime="2026-09-18 20:22:09.248716131 +0000 UTC m=+92.742068708"
Sep 18 20:22:10.895065 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:10.895028    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:22:12.869339 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:12.869297    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:22:12.869794 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:12.869352    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:22:12.874159 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:12.874132    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:22:13.234861 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:13.234830    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:22:13.471146 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:13.471107    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:13.571710 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:13.571629    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:15.614198 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:15.614167    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:15.694141 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:15.694112    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:26.960573 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:26.960543    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:22:26.964304 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:26.964281    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-554d6d74df-lmtzk"
Sep 18 20:22:30.199438 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.199393    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-584fc9568d-gp2fh" podUID="2a9016ff-55c9-42fe-ad91-3d0edab72fd5" containerName="console" containerID="cri-o://cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf" gracePeriod=15
Sep 18 20:22:30.515128 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.515100    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:22:30.632706 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.632658    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-service-ca\") pod \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") "
Sep 18 20:22:30.634689 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.633124    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jlw2\" (UniqueName: \"kubernetes.io/projected/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-kube-api-access-8jlw2\") pod \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") "
Sep 18 20:22:30.634689 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.633536    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-config\") pod \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") "
Sep 18 20:22:30.634689 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.633607    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-oauth-serving-cert\") pod \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") "
Sep 18 20:22:30.634689 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.633649    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-serving-cert\") pod \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") "
Sep 18 20:22:30.634689 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.633687    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-trusted-ca-bundle\") pod \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") "
Sep 18 20:22:30.634689 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.633737    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-oauth-config\") pod \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\" (UID: \"2a9016ff-55c9-42fe-ad91-3d0edab72fd5\") "
Sep 18 20:22:30.635092 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.633053    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-service-ca" (OuterVolumeSpecName: "service-ca") pod "2a9016ff-55c9-42fe-ad91-3d0edab72fd5" (UID: "2a9016ff-55c9-42fe-ad91-3d0edab72fd5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:30.635092 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.634796    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2a9016ff-55c9-42fe-ad91-3d0edab72fd5" (UID: "2a9016ff-55c9-42fe-ad91-3d0edab72fd5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:30.635092 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.635068    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-config" (OuterVolumeSpecName: "console-config") pod "2a9016ff-55c9-42fe-ad91-3d0edab72fd5" (UID: "2a9016ff-55c9-42fe-ad91-3d0edab72fd5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:30.635403 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.635336    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2a9016ff-55c9-42fe-ad91-3d0edab72fd5" (UID: "2a9016ff-55c9-42fe-ad91-3d0edab72fd5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:30.639086 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.639060    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-kube-api-access-8jlw2" (OuterVolumeSpecName: "kube-api-access-8jlw2") pod "2a9016ff-55c9-42fe-ad91-3d0edab72fd5" (UID: "2a9016ff-55c9-42fe-ad91-3d0edab72fd5"). InnerVolumeSpecName "kube-api-access-8jlw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:30.639277 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.639259    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2a9016ff-55c9-42fe-ad91-3d0edab72fd5" (UID: "2a9016ff-55c9-42fe-ad91-3d0edab72fd5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:30.639376 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.639356    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2a9016ff-55c9-42fe-ad91-3d0edab72fd5" (UID: "2a9016ff-55c9-42fe-ad91-3d0edab72fd5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:30.735161 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.735121    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-serving-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.735272 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.735166    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-trusted-ca-bundle\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.735272 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.735183    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-oauth-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.735272 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.735197    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-service-ca\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.735272 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.735211    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jlw2\" (UniqueName: \"kubernetes.io/projected/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-kube-api-access-8jlw2\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.735272 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.735225    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-console-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.735272 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:30.735238    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a9016ff-55c9-42fe-ad91-3d0edab72fd5-oauth-serving-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:22:31.305216 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:31.305175    2557 generic.go:356] "Generic (PLEG): container finished" podID="2a9016ff-55c9-42fe-ad91-3d0edab72fd5" containerID="cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf" exitCode=0
Sep 18 20:22:31.305680 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:31.305260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fc9568d-gp2fh" event={"ID":"2a9016ff-55c9-42fe-ad91-3d0edab72fd5","Type":"ContainerDied","Data":"cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf"}
Sep 18 20:22:31.305680 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:31.305351    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fc9568d-gp2fh" event={"ID":"2a9016ff-55c9-42fe-ad91-3d0edab72fd5","Type":"ContainerDied","Data":"89fbdc838f031576e4091b11505ce1d29bf134f72048c2b507b1b9c68bd3e73d"}
Sep 18 20:22:31.305680 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:31.305383    2557 scope.go:117] "RemoveContainer" containerID="cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf"
Sep 18 20:22:31.305680 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:31.305603    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fc9568d-gp2fh"
Sep 18 20:22:31.333981 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:31.320690    2557 scope.go:117] "RemoveContainer" containerID="cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf"
Sep 18 20:22:31.333981 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:22:31.321090    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf\": container with ID starting with cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf not found: ID does not exist" containerID="cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf"
Sep 18 20:22:31.333981 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:31.321129    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf"} err="failed to get container status \"cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf\": rpc error: code = NotFound desc = could not find container \"cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf\": container with ID starting with cd191d8d1829d073a009085fe1744d0979ae8d47f7be3034b1ba49ead5ea77bf not found: ID does not exist"
Sep 18 20:22:31.333981 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:31.327076    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-584fc9568d-gp2fh"]
Sep 18 20:22:31.333981 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:31.330145    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-584fc9568d-gp2fh"]
Sep 18 20:22:32.722780 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:32.722746    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a9016ff-55c9-42fe-ad91-3d0edab72fd5" path="/var/lib/kubelet/pods/2a9016ff-55c9-42fe-ad91-3d0edab72fd5/volumes"
Sep 18 20:22:36.196156 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:36.196124    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bm5dj"
Sep 18 20:22:58.447775 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:58.447741    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:58.508980 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:22:58.508949    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:01.936886 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.936851    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65dc545545-bwl9t"]
Sep 18 20:23:01.937253 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.937151    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9016ff-55c9-42fe-ad91-3d0edab72fd5" containerName="console"
Sep 18 20:23:01.937253 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.937162    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9016ff-55c9-42fe-ad91-3d0edab72fd5" containerName="console"
Sep 18 20:23:01.937253 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.937209    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9016ff-55c9-42fe-ad91-3d0edab72fd5" containerName="console"
Sep 18 20:23:01.939090 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.939074    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:01.952505 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.952482    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dc545545-bwl9t"]
Sep 18 20:23:01.993785 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.993748    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:01.994140 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.994121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-trusted-ca-bundle\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:01.994198 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.994150    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-config\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:01.994198 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.994175    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nhlp\" (UniqueName: \"kubernetes.io/projected/a19ad20e-8b47-413d-a992-66f8da00b0ae-kube-api-access-9nhlp\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:01.994198 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.994193    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-oauth-config\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:01.994354 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.994248    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-service-ca\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:01.994354 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.994329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-serving-cert\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:01.994449 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:01.994387    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-oauth-serving-cert\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.055270 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.055240    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:02.095636 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.095612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-trusted-ca-bundle\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.095738 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.095644    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-config\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.095738 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.095663    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nhlp\" (UniqueName: \"kubernetes.io/projected/a19ad20e-8b47-413d-a992-66f8da00b0ae-kube-api-access-9nhlp\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.095738 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.095682    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-oauth-config\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.095919 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.095793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-service-ca\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.095919 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.095855    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-serving-cert\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.096072 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.096053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-oauth-serving-cert\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.096411 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.096381    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-config\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.096555 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.096534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-service-ca\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.096687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.096605    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-trusted-ca-bundle\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.096754 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.096705    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-oauth-serving-cert\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.098153 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.098126    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-oauth-config\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.098321 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.098302    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-serving-cert\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.102690 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.102668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nhlp\" (UniqueName: \"kubernetes.io/projected/a19ad20e-8b47-413d-a992-66f8da00b0ae-kube-api-access-9nhlp\") pod \"console-65dc545545-bwl9t\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") " pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.247605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.247531    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:02.518129 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.518058    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dc545545-bwl9t"]
Sep 18 20:23:02.521378 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:23:02.521345    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19ad20e_8b47_413d_a992_66f8da00b0ae.slice/crio-2e95a0844b758bf60257c1457346df2e5691b257f07c8731005160066af7c820 WatchSource:0}: Error finding container 2e95a0844b758bf60257c1457346df2e5691b257f07c8731005160066af7c820: Status 404 returned error can't find the container with id 2e95a0844b758bf60257c1457346df2e5691b257f07c8731005160066af7c820
Sep 18 20:23:02.698259 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.698224    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:23:02.698573 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.698530    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="alertmanager" containerID="cri-o://7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a" gracePeriod=120
Sep 18 20:23:02.698653 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.698596    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy-metric" containerID="cri-o://e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2" gracePeriod=120
Sep 18 20:23:02.698653 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.698618    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="prom-label-proxy" containerID="cri-o://725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e" gracePeriod=120
Sep 18 20:23:02.698750 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.698647    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy" containerID="cri-o://23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516" gracePeriod=120
Sep 18 20:23:02.698750 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.698626    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy-web" containerID="cri-o://d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423" gracePeriod=120
Sep 18 20:23:02.698862 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:02.698646    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="config-reloader" containerID="cri-o://912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45" gracePeriod=120
Sep 18 20:23:03.436977 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.436934    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dc545545-bwl9t" event={"ID":"a19ad20e-8b47-413d-a992-66f8da00b0ae","Type":"ContainerStarted","Data":"df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20"}
Sep 18 20:23:03.437404 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.437025    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dc545545-bwl9t" event={"ID":"a19ad20e-8b47-413d-a992-66f8da00b0ae","Type":"ContainerStarted","Data":"2e95a0844b758bf60257c1457346df2e5691b257f07c8731005160066af7c820"}
Sep 18 20:23:03.440398 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.440371    2557 generic.go:356] "Generic (PLEG): container finished" podID="d81edf6f-0611-4809-b514-323afaabfbb7" containerID="725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e" exitCode=0
Sep 18 20:23:03.440398 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.440393    2557 generic.go:356] "Generic (PLEG): container finished" podID="d81edf6f-0611-4809-b514-323afaabfbb7" containerID="23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516" exitCode=0
Sep 18 20:23:03.440398 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.440399    2557 generic.go:356] "Generic (PLEG): container finished" podID="d81edf6f-0611-4809-b514-323afaabfbb7" containerID="912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45" exitCode=0
Sep 18 20:23:03.440650 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.440406    2557 generic.go:356] "Generic (PLEG): container finished" podID="d81edf6f-0611-4809-b514-323afaabfbb7" containerID="7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a" exitCode=0
Sep 18 20:23:03.440650 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.440442    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerDied","Data":"725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e"}
Sep 18 20:23:03.440650 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.440546    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerDied","Data":"23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516"}
Sep 18 20:23:03.440650 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.440588    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerDied","Data":"912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45"}
Sep 18 20:23:03.440650 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.440631    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerDied","Data":"7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a"}
Sep 18 20:23:03.455697 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.455648    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65dc545545-bwl9t" podStartSLOduration=2.455633929 podStartE2EDuration="2.455633929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:03.454602429 +0000 UTC m=+146.947955008" watchObservedRunningTime="2026-09-18 20:23:03.455633929 +0000 UTC m=+146.948986512"
Sep 18 20:23:03.933857 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:03.933835    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.012506 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012475    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-config-out\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012526    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-cluster-tls-config\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012546    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-main-db\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012576    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-trusted-ca-bundle\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012602    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012623    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012659    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-main-tls\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012994 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012694    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012994 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012734    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-tls-assets\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012994 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012757    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhwt7\" (UniqueName: \"kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-kube-api-access-vhwt7\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012994 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012801    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-web-config\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012994 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012828    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-config-volume\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012994 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012860    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-metrics-client-ca\") pod \"d81edf6f-0611-4809-b514-323afaabfbb7\" (UID: \"d81edf6f-0611-4809-b514-323afaabfbb7\") "
Sep 18 20:23:04.012994 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.012964    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:04.013340 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.013023    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:04.013340 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.013260    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:04.013340 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.013262    2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-main-db\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.013340 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.013293    2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.015763 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.015732    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-config-out" (OuterVolumeSpecName: "config-out") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:04.016985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.016838    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-kube-api-access-vhwt7" (OuterVolumeSpecName: "kube-api-access-vhwt7") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "kube-api-access-vhwt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:04.016985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.016911    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:04.016985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.016963    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-config-volume" (OuterVolumeSpecName: "config-volume") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:04.017194 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.017011    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:04.017194 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.017138    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:04.017915 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.017889    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:04.018021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.017951    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:04.020964 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.020942    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:04.026868 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.026819    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-web-config" (OuterVolumeSpecName: "web-config") pod "d81edf6f-0611-4809-b514-323afaabfbb7" (UID: "d81edf6f-0611-4809-b514-323afaabfbb7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:04.114206 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114185    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-tls-assets\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.114206 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114206    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwt7\" (UniqueName: \"kubernetes.io/projected/d81edf6f-0611-4809-b514-323afaabfbb7-kube-api-access-vhwt7\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.114326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114216    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-web-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.114326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114225    2557 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-config-volume\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.114326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114232    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d81edf6f-0611-4809-b514-323afaabfbb7-metrics-client-ca\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.114326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114239    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d81edf6f-0611-4809-b514-323afaabfbb7-config-out\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.114326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114247    2557 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-cluster-tls-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.114326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114256    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.114326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114265    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.114326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114276    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-main-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.114326 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.114284    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d81edf6f-0611-4809-b514-323afaabfbb7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:04.447837 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.447805    2557 generic.go:356] "Generic (PLEG): container finished" podID="d81edf6f-0611-4809-b514-323afaabfbb7" containerID="e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2" exitCode=0
Sep 18 20:23:04.447837 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.447830    2557 generic.go:356] "Generic (PLEG): container finished" podID="d81edf6f-0611-4809-b514-323afaabfbb7" containerID="d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423" exitCode=0
Sep 18 20:23:04.448287 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.447879    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.448287 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.447886    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerDied","Data":"e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2"}
Sep 18 20:23:04.448287 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.447952    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerDied","Data":"d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423"}
Sep 18 20:23:04.448287 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.447980    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d81edf6f-0611-4809-b514-323afaabfbb7","Type":"ContainerDied","Data":"33f3f067257e9884618e213032b6bad4c5c2032e7ca5750b7b050306beb2754d"}
Sep 18 20:23:04.448287 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.448010    2557 scope.go:117] "RemoveContainer" containerID="725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e"
Sep 18 20:23:04.455922 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.455904    2557 scope.go:117] "RemoveContainer" containerID="e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2"
Sep 18 20:23:04.462837 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.462819    2557 scope.go:117] "RemoveContainer" containerID="23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516"
Sep 18 20:23:04.469581 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.469558    2557 scope.go:117] "RemoveContainer" containerID="d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423"
Sep 18 20:23:04.470615 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.470552    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:23:04.474761 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.474740    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:23:04.476709 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.476690    2557 scope.go:117] "RemoveContainer" containerID="912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45"
Sep 18 20:23:04.483439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.483423    2557 scope.go:117] "RemoveContainer" containerID="7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a"
Sep 18 20:23:04.490121 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.490102    2557 scope.go:117] "RemoveContainer" containerID="4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c"
Sep 18 20:23:04.496670 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.496653    2557 scope.go:117] "RemoveContainer" containerID="725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e"
Sep 18 20:23:04.496928 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:23:04.496908    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e\": container with ID starting with 725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e not found: ID does not exist" containerID="725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e"
Sep 18 20:23:04.496972 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.496939    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e"} err="failed to get container status \"725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e\": rpc error: code = NotFound desc = could not find container \"725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e\": container with ID starting with 725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e not found: ID does not exist"
Sep 18 20:23:04.496972 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.496957    2557 scope.go:117] "RemoveContainer" containerID="e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2"
Sep 18 20:23:04.497205 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:23:04.497186    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2\": container with ID starting with e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2 not found: ID does not exist" containerID="e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2"
Sep 18 20:23:04.497244 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.497212    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2"} err="failed to get container status \"e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2\": rpc error: code = NotFound desc = could not find container \"e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2\": container with ID starting with e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2 not found: ID does not exist"
Sep 18 20:23:04.497244 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.497229    2557 scope.go:117] "RemoveContainer" containerID="23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516"
Sep 18 20:23:04.497437 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:23:04.497422    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516\": container with ID starting with 23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516 not found: ID does not exist" containerID="23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516"
Sep 18 20:23:04.497496 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.497441    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516"} err="failed to get container status \"23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516\": rpc error: code = NotFound desc = could not find container \"23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516\": container with ID starting with 23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516 not found: ID does not exist"
Sep 18 20:23:04.497496 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.497475    2557 scope.go:117] "RemoveContainer" containerID="d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423"
Sep 18 20:23:04.497727 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:23:04.497711    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423\": container with ID starting with d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423 not found: ID does not exist" containerID="d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423"
Sep 18 20:23:04.497777 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.497729    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423"} err="failed to get container status \"d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423\": rpc error: code = NotFound desc = could not find container \"d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423\": container with ID starting with d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423 not found: ID does not exist"
Sep 18 20:23:04.497777 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.497741    2557 scope.go:117] "RemoveContainer" containerID="912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45"
Sep 18 20:23:04.497951 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:23:04.497936    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45\": container with ID starting with 912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45 not found: ID does not exist" containerID="912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45"
Sep 18 20:23:04.497993 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.497953    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45"} err="failed to get container status \"912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45\": rpc error: code = NotFound desc = could not find container \"912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45\": container with ID starting with 912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45 not found: ID does not exist"
Sep 18 20:23:04.497993 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.497965    2557 scope.go:117] "RemoveContainer" containerID="7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a"
Sep 18 20:23:04.498193 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:23:04.498176    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a\": container with ID starting with 7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a not found: ID does not exist" containerID="7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a"
Sep 18 20:23:04.498227 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.498198    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a"} err="failed to get container status \"7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a\": rpc error: code = NotFound desc = could not find container \"7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a\": container with ID starting with 7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a not found: ID does not exist"
Sep 18 20:23:04.498227 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.498215    2557 scope.go:117] "RemoveContainer" containerID="4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c"
Sep 18 20:23:04.498438 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:23:04.498423    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c\": container with ID starting with 4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c not found: ID does not exist" containerID="4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c"
Sep 18 20:23:04.498493 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.498440    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c"} err="failed to get container status \"4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c\": rpc error: code = NotFound desc = could not find container \"4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c\": container with ID starting with 4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c not found: ID does not exist"
Sep 18 20:23:04.498561 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.498548    2557 scope.go:117] "RemoveContainer" containerID="725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e"
Sep 18 20:23:04.498779 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.498761    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e"} err="failed to get container status \"725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e\": rpc error: code = NotFound desc = could not find container \"725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e\": container with ID starting with 725629fbd69d1494db6d1ae615f9846176c38202cc5a312ece943ab07177267e not found: ID does not exist"
Sep 18 20:23:04.498779 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.498778    2557 scope.go:117] "RemoveContainer" containerID="e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2"
Sep 18 20:23:04.498984 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.498968    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2"} err="failed to get container status \"e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2\": rpc error: code = NotFound desc = could not find container \"e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2\": container with ID starting with e6573fdad1fc43d03528a0403d85fcba4979e75af0a401f4258ef4f29abad6a2 not found: ID does not exist"
Sep 18 20:23:04.499026 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.498985    2557 scope.go:117] "RemoveContainer" containerID="23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516"
Sep 18 20:23:04.499198 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.499179    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516"} err="failed to get container status \"23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516\": rpc error: code = NotFound desc = could not find container \"23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516\": container with ID starting with 23d90ad50f590b5e474fe15728bbc01d103645a2862a7dd3c428dcaa3762b516 not found: ID does not exist"
Sep 18 20:23:04.499239 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.499201    2557 scope.go:117] "RemoveContainer" containerID="d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423"
Sep 18 20:23:04.499392 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.499377    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423"} err="failed to get container status \"d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423\": rpc error: code = NotFound desc = could not find container \"d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423\": container with ID starting with d3b15826aa5e3ef59d19cd16b89b12c3a8748a57ecb40d16c72a34f7dad9e423 not found: ID does not exist"
Sep 18 20:23:04.499436 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.499392    2557 scope.go:117] "RemoveContainer" containerID="912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45"
Sep 18 20:23:04.499634 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.499617    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45"} err="failed to get container status \"912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45\": rpc error: code = NotFound desc = could not find container \"912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45\": container with ID starting with 912a42bea5e2a187707f20941265e9ce6aa183b1fd620cfc45cef712b2bb3a45 not found: ID does not exist"
Sep 18 20:23:04.499682 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.499634    2557 scope.go:117] "RemoveContainer" containerID="7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a"
Sep 18 20:23:04.499843 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.499828    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a"} err="failed to get container status \"7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a\": rpc error: code = NotFound desc = could not find container \"7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a\": container with ID starting with 7a1480ce8beef98e35cad7c9298bd51ccc66fdf0803eab758ff630c68badb15a not found: ID does not exist"
Sep 18 20:23:04.499843 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.499842    2557 scope.go:117] "RemoveContainer" containerID="4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c"
Sep 18 20:23:04.500086 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.500069    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c"} err="failed to get container status \"4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c\": rpc error: code = NotFound desc = could not find container \"4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c\": container with ID starting with 4e01a199efbe50415a4a1d9aa3fbaf78e9e23cc59693d55dbe47c472c0708e4c not found: ID does not exist"
Sep 18 20:23:04.511800 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.511777    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:23:04.512048 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512036    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy"
Sep 18 20:23:04.512086 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512049    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy"
Sep 18 20:23:04.512086 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512059    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="config-reloader"
Sep 18 20:23:04.512086 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512065    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="config-reloader"
Sep 18 20:23:04.512086 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512070    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="prom-label-proxy"
Sep 18 20:23:04.512086 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512076    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="prom-label-proxy"
Sep 18 20:23:04.512086 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512084    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy-web"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512089    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy-web"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512099    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="init-config-reloader"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512104    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="init-config-reloader"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512117    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="alertmanager"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512129    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="alertmanager"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512133    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy-metric"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512138    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy-metric"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512184    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="config-reloader"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512190    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="alertmanager"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512195    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512201    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy-web"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512206    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="kube-rbac-proxy-metric"
Sep 18 20:23:04.512256 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.512211    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" containerName="prom-label-proxy"
Sep 18 20:23:04.515605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.515588    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.519730 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.519708    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 20:23:04.519829 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.519736    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m85gr\""
Sep 18 20:23:04.519829 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.519751    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 20:23:04.519829 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.519737    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 20:23:04.519829 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.519781    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 20:23:04.520115 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.520087    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 20:23:04.520115 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.520089    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 20:23:04.520115 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.520108    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 20:23:04.520354 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.520339    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 20:23:04.527263 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.527246    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 20:23:04.550602 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.550575    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:23:04.617100 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617061    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58cea09a-cf80-4b94-83df-7a1487dfd5aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617100 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617099    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617327 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617119    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58cea09a-cf80-4b94-83df-7a1487dfd5aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617327 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617190    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617327 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617219    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617327 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617238    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-web-config\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617327 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617256    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58cea09a-cf80-4b94-83df-7a1487dfd5aa-config-out\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617327 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617275    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617327 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617289    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58cea09a-cf80-4b94-83df-7a1487dfd5aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617327 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617327 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617321    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58cea09a-cf80-4b94-83df-7a1487dfd5aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617679 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617341    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtt5j\" (UniqueName: \"kubernetes.io/projected/58cea09a-cf80-4b94-83df-7a1487dfd5aa-kube-api-access-dtt5j\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.617679 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.617427    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718416 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718333    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718416 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58cea09a-cf80-4b94-83df-7a1487dfd5aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718416 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718416 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718413    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58cea09a-cf80-4b94-83df-7a1487dfd5aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718476    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718516    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-web-config\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58cea09a-cf80-4b94-83df-7a1487dfd5aa-config-out\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718603    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718622    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58cea09a-cf80-4b94-83df-7a1487dfd5aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718644    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718670    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58cea09a-cf80-4b94-83df-7a1487dfd5aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.718771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtt5j\" (UniqueName: \"kubernetes.io/projected/58cea09a-cf80-4b94-83df-7a1487dfd5aa-kube-api-access-dtt5j\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.719210 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.718814    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58cea09a-cf80-4b94-83df-7a1487dfd5aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.719789 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.719759    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58cea09a-cf80-4b94-83df-7a1487dfd5aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.720022 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.719992    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58cea09a-cf80-4b94-83df-7a1487dfd5aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.721665 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.721561    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.722125 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.721683    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.722125 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.721722    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-web-config\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.722125 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.721771    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.722274 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.722129    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58cea09a-cf80-4b94-83df-7a1487dfd5aa-config-out\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.722274 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.722184    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.722619 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.722599    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58cea09a-cf80-4b94-83df-7a1487dfd5aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.725681 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.724262    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.725681 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.724714    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58cea09a-cf80-4b94-83df-7a1487dfd5aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.726921 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.726888    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81edf6f-0611-4809-b514-323afaabfbb7" path="/var/lib/kubelet/pods/d81edf6f-0611-4809-b514-323afaabfbb7/volumes"
Sep 18 20:23:04.727734 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.727708    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtt5j\" (UniqueName: \"kubernetes.io/projected/58cea09a-cf80-4b94-83df-7a1487dfd5aa-kube-api-access-dtt5j\") pod \"alertmanager-main-0\" (UID: \"58cea09a-cf80-4b94-83df-7a1487dfd5aa\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:04.825155 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:04.825122    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:23:05.101230 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:05.101054    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:23:05.105230 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:23:05.105201    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58cea09a_cf80_4b94_83df_7a1487dfd5aa.slice/crio-3a4f45ec8719c6c7b20b3ec50ee849ee41467bbad091ce569e29364ab6555886 WatchSource:0}: Error finding container 3a4f45ec8719c6c7b20b3ec50ee849ee41467bbad091ce569e29364ab6555886: Status 404 returned error can't find the container with id 3a4f45ec8719c6c7b20b3ec50ee849ee41467bbad091ce569e29364ab6555886
Sep 18 20:23:05.454920 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:05.454886    2557 generic.go:356] "Generic (PLEG): container finished" podID="58cea09a-cf80-4b94-83df-7a1487dfd5aa" containerID="be082266ea6e2f851c5d80ddc9f14631c623ec358359e0cf210fcae8774be610" exitCode=0
Sep 18 20:23:05.455359 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:05.454962    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58cea09a-cf80-4b94-83df-7a1487dfd5aa","Type":"ContainerDied","Data":"be082266ea6e2f851c5d80ddc9f14631c623ec358359e0cf210fcae8774be610"}
Sep 18 20:23:05.455359 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:05.455036    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58cea09a-cf80-4b94-83df-7a1487dfd5aa","Type":"ContainerStarted","Data":"3a4f45ec8719c6c7b20b3ec50ee849ee41467bbad091ce569e29364ab6555886"}
Sep 18 20:23:06.463187 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:06.463151    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58cea09a-cf80-4b94-83df-7a1487dfd5aa","Type":"ContainerStarted","Data":"e50f301d7b3798c31b4a97e5efeb5ec8beb1d2e6d515d4e0feb29ed9bc11a3a4"}
Sep 18 20:23:06.463566 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:06.463219    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58cea09a-cf80-4b94-83df-7a1487dfd5aa","Type":"ContainerStarted","Data":"096c7bc14cf37fbc8e7a392cfaaca50b79697db658b55296a355206e50efd59a"}
Sep 18 20:23:06.463566 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:06.463246    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58cea09a-cf80-4b94-83df-7a1487dfd5aa","Type":"ContainerStarted","Data":"c30578633fc241cbc0259fab5bfc27d644570214fdf4bfc7ebe1f65ce3768371"}
Sep 18 20:23:06.463566 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:06.463269    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58cea09a-cf80-4b94-83df-7a1487dfd5aa","Type":"ContainerStarted","Data":"0e3bd6c271de2c43349e30735e01f555828c9ad60586ce62c3713ee6c53d06c6"}
Sep 18 20:23:06.463566 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:06.463298    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58cea09a-cf80-4b94-83df-7a1487dfd5aa","Type":"ContainerStarted","Data":"f384b3787c0e32c43cb0a5ecece8bb86c9f0ca75bf7b287d79954e26d2c88e8a"}
Sep 18 20:23:06.463566 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:06.463320    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58cea09a-cf80-4b94-83df-7a1487dfd5aa","Type":"ContainerStarted","Data":"b621a227433a87dbdf283250aaa81bb18d6c7ffe5243a35e6641b4bf866dfdda"}
Sep 18 20:23:06.489771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:06.489689    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.476877775 podStartE2EDuration="2.48966866s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.790885ms" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:06.48775692 +0000 UTC m=+149.981109497" watchObservedRunningTime="2026-09-18 20:23:06.48966866 +0000 UTC m=+149.983021242"
Sep 18 20:23:10.410273 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:10.410237    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:10.479818 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:10.479763    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:12.248275 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:12.248222    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:12.248667 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:12.248302    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:12.252934 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:12.252911    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:12.490225 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:12.490192    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:23:12.540803 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:12.540719    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86649cfc57-2zdzh"]
Sep 18 20:23:12.720271 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:12.720236    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:12.818316 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:12.818209    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:15.627548 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:15.627509    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:15.726976 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:15.726944    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:37.560655 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.560544    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86649cfc57-2zdzh" podUID="a66eeb85-9624-4396-9c89-341d50626bc6" containerName="console" containerID="cri-o://bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b" gracePeriod=15
Sep 18 20:23:37.797627 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.797598    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:23:37.900835 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.900740    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4wl5\" (UniqueName: \"kubernetes.io/projected/a66eeb85-9624-4396-9c89-341d50626bc6-kube-api-access-h4wl5\") pod \"a66eeb85-9624-4396-9c89-341d50626bc6\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") "
Sep 18 20:23:37.900835 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.900779    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-service-ca\") pod \"a66eeb85-9624-4396-9c89-341d50626bc6\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") "
Sep 18 20:23:37.900835 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.900816    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-console-config\") pod \"a66eeb85-9624-4396-9c89-341d50626bc6\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") "
Sep 18 20:23:37.901074 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.900856    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-trusted-ca-bundle\") pod \"a66eeb85-9624-4396-9c89-341d50626bc6\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") "
Sep 18 20:23:37.901074 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.900886    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-serving-cert\") pod \"a66eeb85-9624-4396-9c89-341d50626bc6\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") "
Sep 18 20:23:37.901074 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.900931    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-oauth-serving-cert\") pod \"a66eeb85-9624-4396-9c89-341d50626bc6\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") "
Sep 18 20:23:37.901074 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.900950    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-oauth-config\") pod \"a66eeb85-9624-4396-9c89-341d50626bc6\" (UID: \"a66eeb85-9624-4396-9c89-341d50626bc6\") "
Sep 18 20:23:37.901303 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.901210    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-service-ca" (OuterVolumeSpecName: "service-ca") pod "a66eeb85-9624-4396-9c89-341d50626bc6" (UID: "a66eeb85-9624-4396-9c89-341d50626bc6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:37.901364 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.901314    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a66eeb85-9624-4396-9c89-341d50626bc6" (UID: "a66eeb85-9624-4396-9c89-341d50626bc6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:37.901364 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.901337    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-console-config" (OuterVolumeSpecName: "console-config") pod "a66eeb85-9624-4396-9c89-341d50626bc6" (UID: "a66eeb85-9624-4396-9c89-341d50626bc6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:37.901496 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.901429    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a66eeb85-9624-4396-9c89-341d50626bc6" (UID: "a66eeb85-9624-4396-9c89-341d50626bc6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:37.903201 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.903169    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a66eeb85-9624-4396-9c89-341d50626bc6" (UID: "a66eeb85-9624-4396-9c89-341d50626bc6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:37.903323 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.903250    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a66eeb85-9624-4396-9c89-341d50626bc6" (UID: "a66eeb85-9624-4396-9c89-341d50626bc6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:37.903323 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:37.903248    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66eeb85-9624-4396-9c89-341d50626bc6-kube-api-access-h4wl5" (OuterVolumeSpecName: "kube-api-access-h4wl5") pod "a66eeb85-9624-4396-9c89-341d50626bc6" (UID: "a66eeb85-9624-4396-9c89-341d50626bc6"). InnerVolumeSpecName "kube-api-access-h4wl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:38.001568 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.001530    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-console-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:38.001568 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.001561    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-trusted-ca-bundle\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:38.001568 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.001572    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-serving-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:38.001568 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.001581    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-oauth-serving-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:38.001849 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.001597    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a66eeb85-9624-4396-9c89-341d50626bc6-console-oauth-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:38.001849 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.001606    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4wl5\" (UniqueName: \"kubernetes.io/projected/a66eeb85-9624-4396-9c89-341d50626bc6-kube-api-access-h4wl5\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:38.001849 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.001617    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a66eeb85-9624-4396-9c89-341d50626bc6-service-ca\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:23:38.587765 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.587731    2557 generic.go:356] "Generic (PLEG): container finished" podID="a66eeb85-9624-4396-9c89-341d50626bc6" containerID="bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b" exitCode=0
Sep 18 20:23:38.588144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.587772    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86649cfc57-2zdzh"
Sep 18 20:23:38.588144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.587771    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86649cfc57-2zdzh" event={"ID":"a66eeb85-9624-4396-9c89-341d50626bc6","Type":"ContainerDied","Data":"bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b"}
Sep 18 20:23:38.588144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.587893    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86649cfc57-2zdzh" event={"ID":"a66eeb85-9624-4396-9c89-341d50626bc6","Type":"ContainerDied","Data":"a09a5f7824d450bdca21fcc8f68059229dc334b83d0ba6d0f8386e4a93e6a02d"}
Sep 18 20:23:38.588144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.587926    2557 scope.go:117] "RemoveContainer" containerID="bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b"
Sep 18 20:23:38.596629 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.596608    2557 scope.go:117] "RemoveContainer" containerID="bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b"
Sep 18 20:23:38.596917 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:23:38.596897    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b\": container with ID starting with bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b not found: ID does not exist" containerID="bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b"
Sep 18 20:23:38.596988 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.596924    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b"} err="failed to get container status \"bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b\": rpc error: code = NotFound desc = could not find container \"bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b\": container with ID starting with bf681b53c32883e8d4c1f3baff25c02770e68a4219c68cae9802a194ba91f46b not found: ID does not exist"
Sep 18 20:23:38.607236 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.607207    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86649cfc57-2zdzh"]
Sep 18 20:23:38.611341 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.611317    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86649cfc57-2zdzh"]
Sep 18 20:23:38.722584 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:38.722551    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a66eeb85-9624-4396-9c89-341d50626bc6" path="/var/lib/kubelet/pods/a66eeb85-9624-4396-9c89-341d50626bc6/volumes"
Sep 18 20:23:48.465383 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:48.465346    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:48.526928 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:48.526896    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:51.296893 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:23:51.296859    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:21.527273 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:21.527236    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:21.627056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:21.627009    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:21.686174 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:21.686130    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:22.727687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.727653    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"]
Sep 18 20:24:22.728063 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.727958    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66eeb85-9624-4396-9c89-341d50626bc6" containerName="console"
Sep 18 20:24:22.728063 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.727970    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66eeb85-9624-4396-9c89-341d50626bc6" containerName="console"
Sep 18 20:24:22.728063 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.728030    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a66eeb85-9624-4396-9c89-341d50626bc6" containerName="console"
Sep 18 20:24:22.730012 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.729996    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:22.732471 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.732433    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5x8sr\""
Sep 18 20:24:22.732573 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.732432    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 20:24:22.732633 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.732439    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 20:24:22.737588 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.737562    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"]
Sep 18 20:24:22.852021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.851988    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqbxv\" (UniqueName: \"kubernetes.io/projected/cd3ce200-0415-4ab2-a4fa-5b91157cf079-kube-api-access-mqbxv\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:22.852193 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.852047    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:22.852193 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.852085    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:22.953431 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.953389    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:22.953431 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.953432    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:22.953668 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.953496    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbxv\" (UniqueName: \"kubernetes.io/projected/cd3ce200-0415-4ab2-a4fa-5b91157cf079-kube-api-access-mqbxv\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:22.953840 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.953822    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:22.953877 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.953855    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:22.962820 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:22.962784    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbxv\" (UniqueName: \"kubernetes.io/projected/cd3ce200-0415-4ab2-a4fa-5b91157cf079-kube-api-access-mqbxv\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:23.040740 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:23.040654    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:23.313698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:23.313623    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"]
Sep 18 20:24:23.316407 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:24:23.316381    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3ce200_0415_4ab2_a4fa_5b91157cf079.slice/crio-a5ac9c6c879cb8aa128e3f533ccb385ae6051c7e2320d8051b35caa633a1a3ab WatchSource:0}: Error finding container a5ac9c6c879cb8aa128e3f533ccb385ae6051c7e2320d8051b35caa633a1a3ab: Status 404 returned error can't find the container with id a5ac9c6c879cb8aa128e3f533ccb385ae6051c7e2320d8051b35caa633a1a3ab
Sep 18 20:24:23.758620 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:23.758583    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d" event={"ID":"cd3ce200-0415-4ab2-a4fa-5b91157cf079","Type":"ContainerStarted","Data":"a5ac9c6c879cb8aa128e3f533ccb385ae6051c7e2320d8051b35caa633a1a3ab"}
Sep 18 20:24:27.860870 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:27.860831    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:29.785855 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:29.785810    2557 generic.go:356] "Generic (PLEG): container finished" podID="cd3ce200-0415-4ab2-a4fa-5b91157cf079" containerID="7e88dc0b3ab4eab96484712eef415c5db6265ec9fbe8d247e9320cb560bbd29d" exitCode=0
Sep 18 20:24:29.786251 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:29.785872    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d" event={"ID":"cd3ce200-0415-4ab2-a4fa-5b91157cf079","Type":"ContainerDied","Data":"7e88dc0b3ab4eab96484712eef415c5db6265ec9fbe8d247e9320cb560bbd29d"}
Sep 18 20:24:32.799642 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:32.799603    2557 generic.go:356] "Generic (PLEG): container finished" podID="cd3ce200-0415-4ab2-a4fa-5b91157cf079" containerID="c46bc325219a43fa63395d1b002f9f3658f41b7cd47a085d9d5e11c59005947e" exitCode=0
Sep 18 20:24:32.800048 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:32.799653    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d" event={"ID":"cd3ce200-0415-4ab2-a4fa-5b91157cf079","Type":"ContainerDied","Data":"c46bc325219a43fa63395d1b002f9f3658f41b7cd47a085d9d5e11c59005947e"}
Sep 18 20:24:39.834951 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:39.834912    2557 generic.go:356] "Generic (PLEG): container finished" podID="cd3ce200-0415-4ab2-a4fa-5b91157cf079" containerID="da2bdd32069c4511c22c9e375f621b241ed4f22f7e579ae6937cbec33ff1aab0" exitCode=0
Sep 18 20:24:39.835325 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:39.834960    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d" event={"ID":"cd3ce200-0415-4ab2-a4fa-5b91157cf079","Type":"ContainerDied","Data":"da2bdd32069c4511c22c9e375f621b241ed4f22f7e579ae6937cbec33ff1aab0"}
Sep 18 20:24:40.963261 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:40.963237    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:41.125849 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.125747    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqbxv\" (UniqueName: \"kubernetes.io/projected/cd3ce200-0415-4ab2-a4fa-5b91157cf079-kube-api-access-mqbxv\") pod \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") "
Sep 18 20:24:41.125849 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.125828    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-util\") pod \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") "
Sep 18 20:24:41.125849 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.125851    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-bundle\") pod \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\" (UID: \"cd3ce200-0415-4ab2-a4fa-5b91157cf079\") "
Sep 18 20:24:41.126517 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.126488    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-bundle" (OuterVolumeSpecName: "bundle") pod "cd3ce200-0415-4ab2-a4fa-5b91157cf079" (UID: "cd3ce200-0415-4ab2-a4fa-5b91157cf079"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:41.128041 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.128015    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3ce200-0415-4ab2-a4fa-5b91157cf079-kube-api-access-mqbxv" (OuterVolumeSpecName: "kube-api-access-mqbxv") pod "cd3ce200-0415-4ab2-a4fa-5b91157cf079" (UID: "cd3ce200-0415-4ab2-a4fa-5b91157cf079"). InnerVolumeSpecName "kube-api-access-mqbxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:41.131854 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.131821    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-util" (OuterVolumeSpecName: "util") pod "cd3ce200-0415-4ab2-a4fa-5b91157cf079" (UID: "cd3ce200-0415-4ab2-a4fa-5b91157cf079"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:41.226670 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.226626    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqbxv\" (UniqueName: \"kubernetes.io/projected/cd3ce200-0415-4ab2-a4fa-5b91157cf079-kube-api-access-mqbxv\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:24:41.226670 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.226662    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-util\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:24:41.226670 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.226672    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd3ce200-0415-4ab2-a4fa-5b91157cf079-bundle\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:24:41.844553 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.844528    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d"
Sep 18 20:24:41.844553 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.844534    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7nf5d" event={"ID":"cd3ce200-0415-4ab2-a4fa-5b91157cf079","Type":"ContainerDied","Data":"a5ac9c6c879cb8aa128e3f533ccb385ae6051c7e2320d8051b35caa633a1a3ab"}
Sep 18 20:24:41.844758 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:41.844576    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5ac9c6c879cb8aa128e3f533ccb385ae6051c7e2320d8051b35caa633a1a3ab"
Sep 18 20:24:44.178416 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:44.178378    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:44.277230 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:44.277196    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:45.835521 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.835486    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"]
Sep 18 20:24:45.835918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.835779    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3ce200-0415-4ab2-a4fa-5b91157cf079" containerName="pull"
Sep 18 20:24:45.835918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.835791    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3ce200-0415-4ab2-a4fa-5b91157cf079" containerName="pull"
Sep 18 20:24:45.835918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.835808    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3ce200-0415-4ab2-a4fa-5b91157cf079" containerName="util"
Sep 18 20:24:45.835918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.835815    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3ce200-0415-4ab2-a4fa-5b91157cf079" containerName="util"
Sep 18 20:24:45.835918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.835826    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3ce200-0415-4ab2-a4fa-5b91157cf079" containerName="extract"
Sep 18 20:24:45.835918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.835832    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3ce200-0415-4ab2-a4fa-5b91157cf079" containerName="extract"
Sep 18 20:24:45.835918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.835889    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3ce200-0415-4ab2-a4fa-5b91157cf079" containerName="extract"
Sep 18 20:24:45.877244 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.877212    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"]
Sep 18 20:24:45.877402 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.877324    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"
Sep 18 20:24:45.882042 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.882016    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 20:24:45.882167 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.882114    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 20:24:45.882271 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.882255    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 20:24:45.882368 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:45.882353    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-rj9p6\""
Sep 18 20:24:46.067814 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:46.067769    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7zwk\" (UniqueName: \"kubernetes.io/projected/3ca6a99e-13c5-4f18-b569-ac2bc82d2388-kube-api-access-v7zwk\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9tzgp\" (UID: \"3ca6a99e-13c5-4f18-b569-ac2bc82d2388\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"
Sep 18 20:24:46.068009 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:46.067825    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3ca6a99e-13c5-4f18-b569-ac2bc82d2388-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9tzgp\" (UID: \"3ca6a99e-13c5-4f18-b569-ac2bc82d2388\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"
Sep 18 20:24:46.168998 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:46.168914    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7zwk\" (UniqueName: \"kubernetes.io/projected/3ca6a99e-13c5-4f18-b569-ac2bc82d2388-kube-api-access-v7zwk\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9tzgp\" (UID: \"3ca6a99e-13c5-4f18-b569-ac2bc82d2388\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"
Sep 18 20:24:46.168998 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:46.168960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3ca6a99e-13c5-4f18-b569-ac2bc82d2388-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9tzgp\" (UID: \"3ca6a99e-13c5-4f18-b569-ac2bc82d2388\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"
Sep 18 20:24:46.171264 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:46.171244    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3ca6a99e-13c5-4f18-b569-ac2bc82d2388-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9tzgp\" (UID: \"3ca6a99e-13c5-4f18-b569-ac2bc82d2388\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"
Sep 18 20:24:46.182349 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:46.180304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7zwk\" (UniqueName: \"kubernetes.io/projected/3ca6a99e-13c5-4f18-b569-ac2bc82d2388-kube-api-access-v7zwk\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9tzgp\" (UID: \"3ca6a99e-13c5-4f18-b569-ac2bc82d2388\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"
Sep 18 20:24:46.186859 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:46.186836    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"
Sep 18 20:24:46.486149 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:46.485981    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"]
Sep 18 20:24:46.489073 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:24:46.489042    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca6a99e_13c5_4f18_b569_ac2bc82d2388.slice/crio-94fbbaae124752ec32400cd90d339db404cce1e2513826bf5c168124375562dd WatchSource:0}: Error finding container 94fbbaae124752ec32400cd90d339db404cce1e2513826bf5c168124375562dd: Status 404 returned error can't find the container with id 94fbbaae124752ec32400cd90d339db404cce1e2513826bf5c168124375562dd
Sep 18 20:24:46.865086 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:46.864998    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp" event={"ID":"3ca6a99e-13c5-4f18-b569-ac2bc82d2388","Type":"ContainerStarted","Data":"94fbbaae124752ec32400cd90d339db404cce1e2513826bf5c168124375562dd"}
Sep 18 20:24:50.554944 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.554905    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:50.885104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.885008    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp" event={"ID":"3ca6a99e-13c5-4f18-b569-ac2bc82d2388","Type":"ContainerStarted","Data":"653da53a704b0fa2b95ca3ca7bb23d7289188614dc659512e22859fdd67e9f76"}
Sep 18 20:24:50.885104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.885070    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"
Sep 18 20:24:50.910014 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.909976    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:50.911483 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.911421    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp" podStartSLOduration=2.333678042 podStartE2EDuration="5.911405661s" totalImagesPullingTime="3.577727619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:50.908614875 +0000 UTC m=+254.401967455" watchObservedRunningTime="2026-09-18 20:24:50.911405661 +0000 UTC m=+254.404758248"
Sep 18 20:24:50.921783 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.921759    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5r7tn"]
Sep 18 20:24:50.944346 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.944310    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5r7tn"]
Sep 18 20:24:50.944570 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.944443    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:50.946785 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.946756    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 20:24:50.946918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.946809    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 18 20:24:50.946987 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:50.946920    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-4wgwv\""
Sep 18 20:24:51.011781 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.011743    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ca4bd82e-1787-439d-8688-e78db41b8c26-cabundle0\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:51.011969 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.011810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-certificates\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:51.011969 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.011840    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79zn\" (UniqueName: \"kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-kube-api-access-d79zn\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:51.064785 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.064746    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:51.112651 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.112616    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ca4bd82e-1787-439d-8688-e78db41b8c26-cabundle0\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:51.112865 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.112675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-certificates\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:51.112865 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.112698    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79zn\" (UniqueName: \"kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-kube-api-access-d79zn\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:51.112865 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.112836    2557 secret.go:281] references non-existent secret key: ca.crt
Sep 18 20:24:51.112865 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.112859    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 20:24:51.113090 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.112869    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-5r7tn: references non-existent secret key: ca.crt
Sep 18 20:24:51.113090 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.112938    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-certificates podName:ca4bd82e-1787-439d-8688-e78db41b8c26 nodeName:}" failed. No retries permitted until 2026-09-18 20:24:51.61291455 +0000 UTC m=+255.106267120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-certificates") pod "keda-operator-579c8958f7-5r7tn" (UID: "ca4bd82e-1787-439d-8688-e78db41b8c26") : references non-existent secret key: ca.crt
Sep 18 20:24:51.113288 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.113269    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ca4bd82e-1787-439d-8688-e78db41b8c26-cabundle0\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:51.118508 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.118466    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"]
Sep 18 20:24:51.127717 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.127689    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79zn\" (UniqueName: \"kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-kube-api-access-d79zn\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:51.146939 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.146869    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:51.150410 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.150378    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 18 20:24:51.157596 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.157566    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"]
Sep 18 20:24:51.188315 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.188281    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:51.213881 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.213848    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-certificates\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:51.213993 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.213915    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p58bf\" (UniqueName: \"kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-kube-api-access-p58bf\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:51.213993 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.213968    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5647e629-716c-443c-9069-a01f16220529-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:51.315268 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.315232    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p58bf\" (UniqueName: \"kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-kube-api-access-p58bf\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:51.315484 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.315290    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5647e629-716c-443c-9069-a01f16220529-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:51.315533 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.315482    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-certificates\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:51.315624 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.315609    2557 secret.go:281] references non-existent secret key: tls.crt
Sep 18 20:24:51.315665 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.315629    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 18 20:24:51.315665 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.315644    2557 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 18 20:24:51.315665 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.315648    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5647e629-716c-443c-9069-a01f16220529-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:51.315665 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.315662    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 18 20:24:51.315782 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.315732    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-certificates podName:5647e629-716c-443c-9069-a01f16220529 nodeName:}" failed. No retries permitted until 2026-09-18 20:24:51.815718146 +0000 UTC m=+255.309070701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-certificates") pod "keda-metrics-apiserver-77857b7b98-hf7kz" (UID: "5647e629-716c-443c-9069-a01f16220529") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 18 20:24:51.323932 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.323905    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p58bf\" (UniqueName: \"kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-kube-api-access-p58bf\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:51.503033 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.502954    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:51.603874 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.603841    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:51.618109 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.618076    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-certificates\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:51.618255 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.618231    2557 secret.go:281] references non-existent secret key: ca.crt
Sep 18 20:24:51.618255 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.618250    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 20:24:51.618332 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.618262    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-5r7tn: references non-existent secret key: ca.crt
Sep 18 20:24:51.618366 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.618332    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-certificates podName:ca4bd82e-1787-439d-8688-e78db41b8c26 nodeName:}" failed. No retries permitted until 2026-09-18 20:24:52.618312066 +0000 UTC m=+256.111664635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-certificates") pod "keda-operator-579c8958f7-5r7tn" (UID: "ca4bd82e-1787-439d-8688-e78db41b8c26") : references non-existent secret key: ca.crt
Sep 18 20:24:51.819236 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.819151    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-certificates\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:51.819392 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.819266    2557 secret.go:281] references non-existent secret key: tls.crt
Sep 18 20:24:51.819392 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.819280    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 18 20:24:51.819392 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.819298    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz: references non-existent secret key: tls.crt
Sep 18 20:24:51.819392 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:51.819359    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-certificates podName:5647e629-716c-443c-9069-a01f16220529 nodeName:}" failed. No retries permitted until 2026-09-18 20:24:52.819344692 +0000 UTC m=+256.312697248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-certificates") pod "keda-metrics-apiserver-77857b7b98-hf7kz" (UID: "5647e629-716c-443c-9069-a01f16220529") : references non-existent secret key: tls.crt
Sep 18 20:24:51.823639 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.823609    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ql5b9"]
Sep 18 20:24:51.844746 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.844708    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ql5b9"]
Sep 18 20:24:51.844922 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.844822    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:24:51.847104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.847080    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 20:24:51.919735 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.919701    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwxq\" (UniqueName: \"kubernetes.io/projected/e244112a-69dd-40d8-b87d-058c741c7571-kube-api-access-hcwxq\") pod \"keda-admission-6c4d864b86-ql5b9\" (UID: \"e244112a-69dd-40d8-b87d-058c741c7571\") " pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:24:51.919907 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:51.919824    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e244112a-69dd-40d8-b87d-058c741c7571-certificates\") pod \"keda-admission-6c4d864b86-ql5b9\" (UID: \"e244112a-69dd-40d8-b87d-058c741c7571\") " pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:24:52.020568 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.020531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwxq\" (UniqueName: \"kubernetes.io/projected/e244112a-69dd-40d8-b87d-058c741c7571-kube-api-access-hcwxq\") pod \"keda-admission-6c4d864b86-ql5b9\" (UID: \"e244112a-69dd-40d8-b87d-058c741c7571\") " pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:24:52.020758 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.020618    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e244112a-69dd-40d8-b87d-058c741c7571-certificates\") pod \"keda-admission-6c4d864b86-ql5b9\" (UID: \"e244112a-69dd-40d8-b87d-058c741c7571\") " pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:24:52.020758 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:52.020724    2557 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 18 20:24:52.020758 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:52.020745    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-ql5b9: secret "keda-admission-webhooks-certs" not found
Sep 18 20:24:52.020875 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:24:52.020794    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e244112a-69dd-40d8-b87d-058c741c7571-certificates podName:e244112a-69dd-40d8-b87d-058c741c7571 nodeName:}" failed. No retries permitted until 2026-09-18 20:24:52.520780601 +0000 UTC m=+256.014133162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e244112a-69dd-40d8-b87d-058c741c7571-certificates") pod "keda-admission-6c4d864b86-ql5b9" (UID: "e244112a-69dd-40d8-b87d-058c741c7571") : secret "keda-admission-webhooks-certs" not found
Sep 18 20:24:52.029571 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.029544    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwxq\" (UniqueName: \"kubernetes.io/projected/e244112a-69dd-40d8-b87d-058c741c7571-kube-api-access-hcwxq\") pod \"keda-admission-6c4d864b86-ql5b9\" (UID: \"e244112a-69dd-40d8-b87d-058c741c7571\") " pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:24:52.046568 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.046539    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:52.145902 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.145819    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:52.525901 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.525855    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e244112a-69dd-40d8-b87d-058c741c7571-certificates\") pod \"keda-admission-6c4d864b86-ql5b9\" (UID: \"e244112a-69dd-40d8-b87d-058c741c7571\") " pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:24:52.528370 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.528350    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e244112a-69dd-40d8-b87d-058c741c7571-certificates\") pod \"keda-admission-6c4d864b86-ql5b9\" (UID: \"e244112a-69dd-40d8-b87d-058c741c7571\") " pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:24:52.626782 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.626736    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-certificates\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:52.629993 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.629964    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca4bd82e-1787-439d-8688-e78db41b8c26-certificates\") pod \"keda-operator-579c8958f7-5r7tn\" (UID: \"ca4bd82e-1787-439d-8688-e78db41b8c26\") " pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:52.688985 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.688946    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:52.757752 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.757713    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:24:52.757930 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.757769    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:52.788155 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.788030    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:52.829131 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.829090    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-certificates\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:52.831966 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.831939    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5647e629-716c-443c-9069-a01f16220529-certificates\") pod \"keda-metrics-apiserver-77857b7b98-hf7kz\" (UID: \"5647e629-716c-443c-9069-a01f16220529\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:52.916088 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.916060    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:52.960318 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:52.960287    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:53.013575 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:53.013325    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:53.037005 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:53.036787    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ql5b9"]
Sep 18 20:24:53.039274 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:53.039166    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5r7tn"]
Sep 18 20:24:53.040721 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:24:53.040679    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode244112a_69dd_40d8_b87d_058c741c7571.slice/crio-4dd2b29a9d9cdbcc7668726553becf60bd9c266de672604a36eb2f94bbe65068 WatchSource:0}: Error finding container 4dd2b29a9d9cdbcc7668726553becf60bd9c266de672604a36eb2f94bbe65068: Status 404 returned error can't find the container with id 4dd2b29a9d9cdbcc7668726553becf60bd9c266de672604a36eb2f94bbe65068
Sep 18 20:24:53.042922 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:24:53.042899    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4bd82e_1787_439d_8688_e78db41b8c26.slice/crio-29eda9ceeb9d742810b84fb1ca96e39c00d8bef6b4db94daf23dd3525649a071 WatchSource:0}: Error finding container 29eda9ceeb9d742810b84fb1ca96e39c00d8bef6b4db94daf23dd3525649a071: Status 404 returned error can't find the container with id 29eda9ceeb9d742810b84fb1ca96e39c00d8bef6b4db94daf23dd3525649a071
Sep 18 20:24:53.231364 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:53.231332    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"]
Sep 18 20:24:53.234365 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:24:53.234332    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5647e629_716c_443c_9069_a01f16220529.slice/crio-18a84915959301ef98a4c71f8384edf25853bab8aa4d20a457d3c40acd1d956a WatchSource:0}: Error finding container 18a84915959301ef98a4c71f8384edf25853bab8aa4d20a457d3c40acd1d956a: Status 404 returned error can't find the container with id 18a84915959301ef98a4c71f8384edf25853bab8aa4d20a457d3c40acd1d956a
Sep 18 20:24:53.899110 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:53.899067    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz" event={"ID":"5647e629-716c-443c-9069-a01f16220529","Type":"ContainerStarted","Data":"18a84915959301ef98a4c71f8384edf25853bab8aa4d20a457d3c40acd1d956a"}
Sep 18 20:24:53.900273 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:53.900245    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ql5b9" event={"ID":"e244112a-69dd-40d8-b87d-058c741c7571","Type":"ContainerStarted","Data":"4dd2b29a9d9cdbcc7668726553becf60bd9c266de672604a36eb2f94bbe65068"}
Sep 18 20:24:53.901334 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:53.901313    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5r7tn" event={"ID":"ca4bd82e-1787-439d-8688-e78db41b8c26","Type":"ContainerStarted","Data":"29eda9ceeb9d742810b84fb1ca96e39c00d8bef6b4db94daf23dd3525649a071"}
Sep 18 20:24:55.913475 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:55.913425    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ql5b9" event={"ID":"e244112a-69dd-40d8-b87d-058c741c7571","Type":"ContainerStarted","Data":"3f00e90803d70133d641d4e830c494fbd1a25e3c168ba2b966381a5baf416d77"}
Sep 18 20:24:55.913973 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:55.913861    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:24:55.929966 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:55.929566    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-ql5b9" podStartSLOduration=3.049208982 podStartE2EDuration="4.929541639s" totalImagesPullingTime="1.880332657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:55.929356667 +0000 UTC m=+259.422709246" watchObservedRunningTime="2026-09-18 20:24:55.929541639 +0000 UTC m=+259.422894221"
Sep 18 20:24:57.925316 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:57.925215    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5r7tn" event={"ID":"ca4bd82e-1787-439d-8688-e78db41b8c26","Type":"ContainerStarted","Data":"2ae9bb21f09632739098326f88954871b93a8975efe2b6ca4e2930b64023b1f8"}
Sep 18 20:24:57.925316 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:57.925288    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:24:57.926809 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:57.926779    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz" event={"ID":"5647e629-716c-443c-9069-a01f16220529","Type":"ContainerStarted","Data":"c6f9cdcedc7e36c12a8a4b5796bac76a6b40c98dbed76d3811c643d816748247"}
Sep 18 20:24:57.926931 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:57.926832    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:24:57.942283 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:57.942234    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-5r7tn" podStartSLOduration=3.430821922 podStartE2EDuration="7.942217874s" totalImagesPullingTime="4.511395952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:57.940681653 +0000 UTC m=+261.434034228" watchObservedRunningTime="2026-09-18 20:24:57.942217874 +0000 UTC m=+261.435570452"
Sep 18 20:24:57.955981 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:24:57.955935    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz" podStartSLOduration=2.636171653 podStartE2EDuration="6.955919757s" totalImagesPullingTime="4.319748104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:57.955073224 +0000 UTC m=+261.448425800" watchObservedRunningTime="2026-09-18 20:24:57.955919757 +0000 UTC m=+261.449272335"
Sep 18 20:25:08.937905 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:25:08.937877    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hf7kz"
Sep 18 20:25:11.892095 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:25:11.892064    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9tzgp"
Sep 18 20:25:16.921016 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:25:16.920977    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-ql5b9"
Sep 18 20:25:18.933736 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:25:18.933700    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-5r7tn"
Sep 18 20:25:38.608076 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:25:38.608041    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:25:38.610337 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:25:38.610307    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:25:38.611041 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:25:38.611021    2557 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 20:25:42.961441 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:25:42.961403    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:44.547585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:25:44.547548    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:09.546716 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.546672    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:09.610108 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.610077    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-zbn9x"]
Sep 18 20:26:09.612836 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.612818    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:09.615229 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.615209    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 18 20:26:09.615786 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.615765    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 20:26:09.615875 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.615795    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 20:26:09.616081 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.616064    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-4pm9s\""
Sep 18 20:26:09.625153 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.625134    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-zbn9x"]
Sep 18 20:26:09.652813 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.652785    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:09.660612 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.660591    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-vwhkk"]
Sep 18 20:26:09.663018 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.662998    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vwhkk"
Sep 18 20:26:09.665139 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.665120    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 20:26:09.665234 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.665173    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wgp8j\""
Sep 18 20:26:09.673605 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.673584    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-cert\") pod \"kserve-controller-manager-7dcf4bff6-zbn9x\" (UID: \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\") " pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:09.673703 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.673689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qtb\" (UniqueName: \"kubernetes.io/projected/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-kube-api-access-82qtb\") pod \"kserve-controller-manager-7dcf4bff6-zbn9x\" (UID: \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\") " pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:09.674958 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.674939    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vwhkk"]
Sep 18 20:26:09.751403 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.751376    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:09.774534 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.774511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d-data\") pod \"seaweedfs-64568bcd49-vwhkk\" (UID: \"5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d\") " pod="kserve/seaweedfs-64568bcd49-vwhkk"
Sep 18 20:26:09.774630 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.774598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qtb\" (UniqueName: \"kubernetes.io/projected/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-kube-api-access-82qtb\") pod \"kserve-controller-manager-7dcf4bff6-zbn9x\" (UID: \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\") " pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:09.774630 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.774622    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62bx\" (UniqueName: \"kubernetes.io/projected/5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d-kube-api-access-c62bx\") pod \"seaweedfs-64568bcd49-vwhkk\" (UID: \"5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d\") " pod="kserve/seaweedfs-64568bcd49-vwhkk"
Sep 18 20:26:09.774704 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.774649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-cert\") pod \"kserve-controller-manager-7dcf4bff6-zbn9x\" (UID: \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\") " pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:09.774745 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:26:09.774730    2557 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 18 20:26:09.774787 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:26:09.774776    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-cert podName:65e7f9fe-30c3-4ee9-9de5-3f720e09559b nodeName:}" failed. No retries permitted until 2026-09-18 20:26:10.274763319 +0000 UTC m=+333.768115879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-cert") pod "kserve-controller-manager-7dcf4bff6-zbn9x" (UID: "65e7f9fe-30c3-4ee9-9de5-3f720e09559b") : secret "kserve-webhook-server-cert" not found
Sep 18 20:26:09.784671 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.784645    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qtb\" (UniqueName: \"kubernetes.io/projected/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-kube-api-access-82qtb\") pod \"kserve-controller-manager-7dcf4bff6-zbn9x\" (UID: \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\") " pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:09.864979 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.864894    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:09.875013 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.874991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d-data\") pod \"seaweedfs-64568bcd49-vwhkk\" (UID: \"5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d\") " pod="kserve/seaweedfs-64568bcd49-vwhkk"
Sep 18 20:26:09.875070 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.875056    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62bx\" (UniqueName: \"kubernetes.io/projected/5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d-kube-api-access-c62bx\") pod \"seaweedfs-64568bcd49-vwhkk\" (UID: \"5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d\") " pod="kserve/seaweedfs-64568bcd49-vwhkk"
Sep 18 20:26:09.875361 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.875346    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d-data\") pod \"seaweedfs-64568bcd49-vwhkk\" (UID: \"5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d\") " pod="kserve/seaweedfs-64568bcd49-vwhkk"
Sep 18 20:26:09.882588 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.882564    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62bx\" (UniqueName: \"kubernetes.io/projected/5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d-kube-api-access-c62bx\") pod \"seaweedfs-64568bcd49-vwhkk\" (UID: \"5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d\") " pod="kserve/seaweedfs-64568bcd49-vwhkk"
Sep 18 20:26:09.971977 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.971930    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vwhkk"
Sep 18 20:26:09.998096 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:09.998064    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:10.175639 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:10.175556    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:10.240898 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:10.240869    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vwhkk"]
Sep 18 20:26:10.243824 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:26:10.243795    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0d3150_5c7b_4d0e_87cf_754f5f56cd9d.slice/crio-8382cf7eed9a0b61f411427998938c109467eebdd9efdfbc6440fc0c144dc072 WatchSource:0}: Error finding container 8382cf7eed9a0b61f411427998938c109467eebdd9efdfbc6440fc0c144dc072: Status 404 returned error can't find the container with id 8382cf7eed9a0b61f411427998938c109467eebdd9efdfbc6440fc0c144dc072
Sep 18 20:26:10.244472 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:10.244434    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:10.245223 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:10.245200    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:26:10.266133 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:10.266111    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vwhkk" event={"ID":"5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d","Type":"ContainerStarted","Data":"8382cf7eed9a0b61f411427998938c109467eebdd9efdfbc6440fc0c144dc072"}
Sep 18 20:26:10.278491 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:10.278467    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-cert\") pod \"kserve-controller-manager-7dcf4bff6-zbn9x\" (UID: \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\") " pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:10.280749 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:10.280708    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-cert\") pod \"kserve-controller-manager-7dcf4bff6-zbn9x\" (UID: \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\") " pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:10.313262 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:10.313233    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:10.522695 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:10.522673    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:10.791007 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:10.790948    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-zbn9x"]
Sep 18 20:26:10.794125 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:26:10.794098    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e7f9fe_30c3_4ee9_9de5_3f720e09559b.slice/crio-3835f1f74aeb960bec786d4cfd379c26e309193057b4bc09138faf4c7d612e3e WatchSource:0}: Error finding container 3835f1f74aeb960bec786d4cfd379c26e309193057b4bc09138faf4c7d612e3e: Status 404 returned error can't find the container with id 3835f1f74aeb960bec786d4cfd379c26e309193057b4bc09138faf4c7d612e3e
Sep 18 20:26:11.273014 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:11.272976    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x" event={"ID":"65e7f9fe-30c3-4ee9-9de5-3f720e09559b","Type":"ContainerStarted","Data":"3835f1f74aeb960bec786d4cfd379c26e309193057b4bc09138faf4c7d612e3e"}
Sep 18 20:26:15.294853 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:15.294806    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x" event={"ID":"65e7f9fe-30c3-4ee9-9de5-3f720e09559b","Type":"ContainerStarted","Data":"e03c84a3768638bf41623318f225366362f5d33b36653b3ec5b0eb9e833cebbb"}
Sep 18 20:26:15.295292 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:15.294979    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:15.296225 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:15.296204    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vwhkk" event={"ID":"5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d","Type":"ContainerStarted","Data":"cc0d07c484495b6acda7a027ae8a7dbeec94645cb2891ec17da39ef4a4fd1b94"}
Sep 18 20:26:15.296321 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:15.296246    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-vwhkk"
Sep 18 20:26:15.312224 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:15.312181    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x" podStartSLOduration=2.603558009 podStartE2EDuration="6.312168713s" totalImagesPullingTime="3.708610704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:15.310980793 +0000 UTC m=+338.804333373" watchObservedRunningTime="2026-09-18 20:26:15.312168713 +0000 UTC m=+338.805521292"
Sep 18 20:26:15.328991 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:15.328946    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-vwhkk" podStartSLOduration=2.021963137 podStartE2EDuration="6.328934335s" totalImagesPullingTime="4.306971198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:15.327878491 +0000 UTC m=+338.821231082" watchObservedRunningTime="2026-09-18 20:26:15.328934335 +0000 UTC m=+338.822286912"
Sep 18 20:26:21.302697 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:21.302622    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-vwhkk"
Sep 18 20:26:46.242173 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.242133    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bb849ff87-zjffq"]
Sep 18 20:26:46.247613 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.247589    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.279841 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.279809    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb849ff87-zjffq"]
Sep 18 20:26:46.285881 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.285851    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwjhd\" (UniqueName: \"kubernetes.io/projected/ce16887f-17ab-41c1-9f4a-65825c10cce8-kube-api-access-qwjhd\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.286023 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.285892    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce16887f-17ab-41c1-9f4a-65825c10cce8-console-oauth-config\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.286023 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.285914    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce16887f-17ab-41c1-9f4a-65825c10cce8-console-serving-cert\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.286023 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.285961    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-service-ca\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.286023 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.285990    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-console-config\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.286023 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.286015    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-oauth-serving-cert\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.286232 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.286052    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-trusted-ca-bundle\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.305827 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.305794    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:46.386511 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.386448    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-service-ca\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.386707 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.386527    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-console-config\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.386707 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.386551    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-oauth-serving-cert\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.386707 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.386588    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-trusted-ca-bundle\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.386707 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.386643    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwjhd\" (UniqueName: \"kubernetes.io/projected/ce16887f-17ab-41c1-9f4a-65825c10cce8-kube-api-access-qwjhd\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.386707 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.386673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce16887f-17ab-41c1-9f4a-65825c10cce8-console-oauth-config\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.386707 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.386708    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce16887f-17ab-41c1-9f4a-65825c10cce8-console-serving-cert\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.387330 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.387295    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-service-ca\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.387485 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.387412    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-console-config\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.387554 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.387477    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-oauth-serving-cert\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.388068 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.388043    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce16887f-17ab-41c1-9f4a-65825c10cce8-trusted-ca-bundle\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.389337 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.389314    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce16887f-17ab-41c1-9f4a-65825c10cce8-console-serving-cert\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.389438 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.389386    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce16887f-17ab-41c1-9f4a-65825c10cce8-console-oauth-config\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.408341 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.408304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwjhd\" (UniqueName: \"kubernetes.io/projected/ce16887f-17ab-41c1-9f4a-65825c10cce8-kube-api-access-qwjhd\") pod \"console-bb849ff87-zjffq\" (UID: \"ce16887f-17ab-41c1-9f4a-65825c10cce8\") " pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.432448 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.432413    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:46.528212 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.528131    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:46.556362 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.556322    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:46.627870 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.627827    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:46.833785 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.833749    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb849ff87-zjffq"]
Sep 18 20:26:46.836501 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:26:46.836468    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce16887f_17ab_41c1_9f4a_65825c10cce8.slice/crio-3d54f556024f2d788c153ffa2ec26fb17eb10554623f45b6fc2415e636c93a3e WatchSource:0}: Error finding container 3d54f556024f2d788c153ffa2ec26fb17eb10554623f45b6fc2415e636c93a3e: Status 404 returned error can't find the container with id 3d54f556024f2d788c153ffa2ec26fb17eb10554623f45b6fc2415e636c93a3e
Sep 18 20:26:46.960081 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:46.960045    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:47.166986 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:47.166896    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:47.439295 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:47.439261    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb849ff87-zjffq" event={"ID":"ce16887f-17ab-41c1-9f4a-65825c10cce8","Type":"ContainerStarted","Data":"9c6b418a3ba1df4280567bf14fa23e46c6645a2525661a23feecc18b0a1fa9a2"}
Sep 18 20:26:47.439705 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:47.439309    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb849ff87-zjffq" event={"ID":"ce16887f-17ab-41c1-9f4a-65825c10cce8","Type":"ContainerStarted","Data":"3d54f556024f2d788c153ffa2ec26fb17eb10554623f45b6fc2415e636c93a3e"}
Sep 18 20:26:47.462888 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:47.462832    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bb849ff87-zjffq" podStartSLOduration=1.462815857 podStartE2EDuration="1.462815857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:47.462187892 +0000 UTC m=+370.955540461" watchObservedRunningTime="2026-09-18 20:26:47.462815857 +0000 UTC m=+370.956168495"
Sep 18 20:26:54.336621 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.336581    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-zbn9x"]
Sep 18 20:26:54.337043 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.336774    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x" podUID="65e7f9fe-30c3-4ee9-9de5-3f720e09559b" containerName="manager" containerID="cri-o://e03c84a3768638bf41623318f225366362f5d33b36653b3ec5b0eb9e833cebbb" gracePeriod=10
Sep 18 20:26:54.366845 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.366809    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-4g7gd"]
Sep 18 20:26:54.370207 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.370191    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd"
Sep 18 20:26:54.380662 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.380626    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-4g7gd"]
Sep 18 20:26:54.459984 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.459951    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0abcd90c-ff2c-4f0a-a738-9989825e32f0-cert\") pod \"kserve-controller-manager-7dcf4bff6-4g7gd\" (UID: \"0abcd90c-ff2c-4f0a-a738-9989825e32f0\") " pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd"
Sep 18 20:26:54.460119 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.460096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hx6v\" (UniqueName: \"kubernetes.io/projected/0abcd90c-ff2c-4f0a-a738-9989825e32f0-kube-api-access-4hx6v\") pod \"kserve-controller-manager-7dcf4bff6-4g7gd\" (UID: \"0abcd90c-ff2c-4f0a-a738-9989825e32f0\") " pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd"
Sep 18 20:26:54.482197 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.482154    2557 generic.go:356] "Generic (PLEG): container finished" podID="65e7f9fe-30c3-4ee9-9de5-3f720e09559b" containerID="e03c84a3768638bf41623318f225366362f5d33b36653b3ec5b0eb9e833cebbb" exitCode=0
Sep 18 20:26:54.482404 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.482207    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x" event={"ID":"65e7f9fe-30c3-4ee9-9de5-3f720e09559b","Type":"ContainerDied","Data":"e03c84a3768638bf41623318f225366362f5d33b36653b3ec5b0eb9e833cebbb"}
Sep 18 20:26:54.561282 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.561256    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hx6v\" (UniqueName: \"kubernetes.io/projected/0abcd90c-ff2c-4f0a-a738-9989825e32f0-kube-api-access-4hx6v\") pod \"kserve-controller-manager-7dcf4bff6-4g7gd\" (UID: \"0abcd90c-ff2c-4f0a-a738-9989825e32f0\") " pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd"
Sep 18 20:26:54.561406 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.561303    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0abcd90c-ff2c-4f0a-a738-9989825e32f0-cert\") pod \"kserve-controller-manager-7dcf4bff6-4g7gd\" (UID: \"0abcd90c-ff2c-4f0a-a738-9989825e32f0\") " pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd"
Sep 18 20:26:54.563489 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.563443    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0abcd90c-ff2c-4f0a-a738-9989825e32f0-cert\") pod \"kserve-controller-manager-7dcf4bff6-4g7gd\" (UID: \"0abcd90c-ff2c-4f0a-a738-9989825e32f0\") " pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd"
Sep 18 20:26:54.569226 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.569198    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hx6v\" (UniqueName: \"kubernetes.io/projected/0abcd90c-ff2c-4f0a-a738-9989825e32f0-kube-api-access-4hx6v\") pod \"kserve-controller-manager-7dcf4bff6-4g7gd\" (UID: \"0abcd90c-ff2c-4f0a-a738-9989825e32f0\") " pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd"
Sep 18 20:26:54.586084 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.586059    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:54.662675 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.662589    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-cert\") pod \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\" (UID: \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\") "
Sep 18 20:26:54.662839 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.662698    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82qtb\" (UniqueName: \"kubernetes.io/projected/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-kube-api-access-82qtb\") pod \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\" (UID: \"65e7f9fe-30c3-4ee9-9de5-3f720e09559b\") "
Sep 18 20:26:54.664778 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.664749    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-kube-api-access-82qtb" (OuterVolumeSpecName: "kube-api-access-82qtb") pod "65e7f9fe-30c3-4ee9-9de5-3f720e09559b" (UID: "65e7f9fe-30c3-4ee9-9de5-3f720e09559b"). InnerVolumeSpecName "kube-api-access-82qtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:54.664881 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.664851    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-cert" (OuterVolumeSpecName: "cert") pod "65e7f9fe-30c3-4ee9-9de5-3f720e09559b" (UID: "65e7f9fe-30c3-4ee9-9de5-3f720e09559b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:26:54.718331 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.718286    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd"
Sep 18 20:26:54.764209 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.764175    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qtb\" (UniqueName: \"kubernetes.io/projected/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-kube-api-access-82qtb\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:26:54.764209 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.764206    2557 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65e7f9fe-30c3-4ee9-9de5-3f720e09559b-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:26:54.991261 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:54.991235    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-4g7gd"]
Sep 18 20:26:54.993573 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:26:54.993550    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abcd90c_ff2c_4f0a_a738_9989825e32f0.slice/crio-36cca039616c2deac85a7dd803b38c0f0d7b6f8da9b0d61e667dc5bd028cfc6a WatchSource:0}: Error finding container 36cca039616c2deac85a7dd803b38c0f0d7b6f8da9b0d61e667dc5bd028cfc6a: Status 404 returned error can't find the container with id 36cca039616c2deac85a7dd803b38c0f0d7b6f8da9b0d61e667dc5bd028cfc6a
Sep 18 20:26:55.487844 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:55.487818    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x"
Sep 18 20:26:55.488302 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:55.487850    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-zbn9x" event={"ID":"65e7f9fe-30c3-4ee9-9de5-3f720e09559b","Type":"ContainerDied","Data":"3835f1f74aeb960bec786d4cfd379c26e309193057b4bc09138faf4c7d612e3e"}
Sep 18 20:26:55.488302 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:55.487919    2557 scope.go:117] "RemoveContainer" containerID="e03c84a3768638bf41623318f225366362f5d33b36653b3ec5b0eb9e833cebbb"
Sep 18 20:26:55.489672 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:55.489641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd" event={"ID":"0abcd90c-ff2c-4f0a-a738-9989825e32f0","Type":"ContainerStarted","Data":"fe352376b9d1462baeffcdb572aac9ed9351d0f1abab558c31ef1388fe3a4cf3"}
Sep 18 20:26:55.489797 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:55.489699    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd" event={"ID":"0abcd90c-ff2c-4f0a-a738-9989825e32f0","Type":"ContainerStarted","Data":"36cca039616c2deac85a7dd803b38c0f0d7b6f8da9b0d61e667dc5bd028cfc6a"}
Sep 18 20:26:55.489797 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:55.489728    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd"
Sep 18 20:26:55.504839 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:55.504810    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-zbn9x"]
Sep 18 20:26:55.508282 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:55.508259    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-zbn9x"]
Sep 18 20:26:55.524992 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:55.524939    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd" podStartSLOduration=1.124985948 podStartE2EDuration="1.524924453s" totalImagesPullingTime="399.938505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:55.523140104 +0000 UTC m=+379.016492684" watchObservedRunningTime="2026-09-18 20:26:55.524924453 +0000 UTC m=+379.018277031"
Sep 18 20:26:56.557193 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:56.557156    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:56.557618 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:56.557216    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:56.562161 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:56.562137    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:56.722473 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:56.722412    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e7f9fe-30c3-4ee9-9de5-3f720e09559b" path="/var/lib/kubelet/pods/65e7f9fe-30c3-4ee9-9de5-3f720e09559b/volumes"
Sep 18 20:26:57.506661 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:57.506627    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bb849ff87-zjffq"
Sep 18 20:26:57.563397 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:57.563364    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dc545545-bwl9t"]
Sep 18 20:26:57.673176 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:57.673145    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:57.774285 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:57.774207    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:57.875442 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:26:57.875409    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:22.584321 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.584281    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65dc545545-bwl9t" podUID="a19ad20e-8b47-413d-a992-66f8da00b0ae" containerName="console" containerID="cri-o://df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20" gracePeriod=15
Sep 18 20:27:22.832299 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.832277    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:27:22.903830 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.903761    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-serving-cert\") pod \"a19ad20e-8b47-413d-a992-66f8da00b0ae\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") "
Sep 18 20:27:22.903946 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.903843    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-oauth-config\") pod \"a19ad20e-8b47-413d-a992-66f8da00b0ae\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") "
Sep 18 20:27:22.903946 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.903883    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-config\") pod \"a19ad20e-8b47-413d-a992-66f8da00b0ae\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") "
Sep 18 20:27:22.903946 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.903920    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nhlp\" (UniqueName: \"kubernetes.io/projected/a19ad20e-8b47-413d-a992-66f8da00b0ae-kube-api-access-9nhlp\") pod \"a19ad20e-8b47-413d-a992-66f8da00b0ae\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") "
Sep 18 20:27:22.904059 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.903972    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-service-ca\") pod \"a19ad20e-8b47-413d-a992-66f8da00b0ae\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") "
Sep 18 20:27:22.904059 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.904030    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-trusted-ca-bundle\") pod \"a19ad20e-8b47-413d-a992-66f8da00b0ae\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") "
Sep 18 20:27:22.904161 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.904061    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-oauth-serving-cert\") pod \"a19ad20e-8b47-413d-a992-66f8da00b0ae\" (UID: \"a19ad20e-8b47-413d-a992-66f8da00b0ae\") "
Sep 18 20:27:22.904397 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.904368    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-config" (OuterVolumeSpecName: "console-config") pod "a19ad20e-8b47-413d-a992-66f8da00b0ae" (UID: "a19ad20e-8b47-413d-a992-66f8da00b0ae"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:27:22.904485 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.904393    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-service-ca" (OuterVolumeSpecName: "service-ca") pod "a19ad20e-8b47-413d-a992-66f8da00b0ae" (UID: "a19ad20e-8b47-413d-a992-66f8da00b0ae"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:27:22.904485 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.904435    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a19ad20e-8b47-413d-a992-66f8da00b0ae" (UID: "a19ad20e-8b47-413d-a992-66f8da00b0ae"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:27:22.904575 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.904532    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a19ad20e-8b47-413d-a992-66f8da00b0ae" (UID: "a19ad20e-8b47-413d-a992-66f8da00b0ae"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:27:22.905927 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.905898    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19ad20e-8b47-413d-a992-66f8da00b0ae-kube-api-access-9nhlp" (OuterVolumeSpecName: "kube-api-access-9nhlp") pod "a19ad20e-8b47-413d-a992-66f8da00b0ae" (UID: "a19ad20e-8b47-413d-a992-66f8da00b0ae"). InnerVolumeSpecName "kube-api-access-9nhlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:27:22.906039 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.905933    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a19ad20e-8b47-413d-a992-66f8da00b0ae" (UID: "a19ad20e-8b47-413d-a992-66f8da00b0ae"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:27:22.906039 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:22.905971    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a19ad20e-8b47-413d-a992-66f8da00b0ae" (UID: "a19ad20e-8b47-413d-a992-66f8da00b0ae"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:27:23.005600 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.005575    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-service-ca\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:27:23.005600 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.005595    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-trusted-ca-bundle\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:27:23.005600 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.005604    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-oauth-serving-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:27:23.005782 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.005613    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-serving-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:27:23.005782 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.005621    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-oauth-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:27:23.005782 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.005636    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a19ad20e-8b47-413d-a992-66f8da00b0ae-console-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:27:23.005782 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.005646    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nhlp\" (UniqueName: \"kubernetes.io/projected/a19ad20e-8b47-413d-a992-66f8da00b0ae-kube-api-access-9nhlp\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:27:23.614797 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.614766    2557 generic.go:356] "Generic (PLEG): container finished" podID="a19ad20e-8b47-413d-a992-66f8da00b0ae" containerID="df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20" exitCode=0
Sep 18 20:27:23.615144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.614805    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dc545545-bwl9t"
Sep 18 20:27:23.615144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.614851    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dc545545-bwl9t" event={"ID":"a19ad20e-8b47-413d-a992-66f8da00b0ae","Type":"ContainerDied","Data":"df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20"}
Sep 18 20:27:23.615144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.614900    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dc545545-bwl9t" event={"ID":"a19ad20e-8b47-413d-a992-66f8da00b0ae","Type":"ContainerDied","Data":"2e95a0844b758bf60257c1457346df2e5691b257f07c8731005160066af7c820"}
Sep 18 20:27:23.615144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.614921    2557 scope.go:117] "RemoveContainer" containerID="df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20"
Sep 18 20:27:23.624156 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.624139    2557 scope.go:117] "RemoveContainer" containerID="df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20"
Sep 18 20:27:23.624418 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:27:23.624402    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20\": container with ID starting with df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20 not found: ID does not exist" containerID="df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20"
Sep 18 20:27:23.624481 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.624426    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20"} err="failed to get container status \"df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20\": rpc error: code = NotFound desc = could not find container \"df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20\": container with ID starting with df56df73419683cbd135b94b62d3fe27b84158859b670b94b0cea9f84701da20 not found: ID does not exist"
Sep 18 20:27:23.636399 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.636376    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dc545545-bwl9t"]
Sep 18 20:27:23.640465 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:23.640436    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65dc545545-bwl9t"]
Sep 18 20:27:24.721848 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:24.721817    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a19ad20e-8b47-413d-a992-66f8da00b0ae" path="/var/lib/kubelet/pods/a19ad20e-8b47-413d-a992-66f8da00b0ae/volumes"
Sep 18 20:27:26.502603 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:26.502573    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7dcf4bff6-4g7gd"
Sep 18 20:27:27.787127 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.787100    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:27.834488 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.834443    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-9sjtg"]
Sep 18 20:27:27.834783 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.834770    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e7f9fe-30c3-4ee9-9de5-3f720e09559b" containerName="manager"
Sep 18 20:27:27.834823 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.834784    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e7f9fe-30c3-4ee9-9de5-3f720e09559b" containerName="manager"
Sep 18 20:27:27.834823 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.834796    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19ad20e-8b47-413d-a992-66f8da00b0ae" containerName="console"
Sep 18 20:27:27.834823 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.834801    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19ad20e-8b47-413d-a992-66f8da00b0ae" containerName="console"
Sep 18 20:27:27.834916 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.834862    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19ad20e-8b47-413d-a992-66f8da00b0ae" containerName="console"
Sep 18 20:27:27.834916 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.834872    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e7f9fe-30c3-4ee9-9de5-3f720e09559b" containerName="manager"
Sep 18 20:27:27.839237 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.839219    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-9sjtg"
Sep 18 20:27:27.846999 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.846982    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-n2qmz\""
Sep 18 20:27:27.847359 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.847346    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 18 20:27:27.860885 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.860864    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-fsngc"]
Sep 18 20:27:27.864090 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.864067    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-9sjtg"]
Sep 18 20:27:27.864191 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.864154    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:27.866776 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.866757    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-kkx7g\""
Sep 18 20:27:27.867682 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.867665    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 20:27:27.869826 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.869806    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 20:27:27.888739 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.888715    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-fsngc"]
Sep 18 20:27:27.909772 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.909747    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:27.940130 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.940106    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585d62ef-4d14-4ef9-8c75-c7c64fcd81dd-tls-certs\") pod \"model-serving-api-7b954ffc95-9sjtg\" (UID: \"585d62ef-4d14-4ef9-8c75-c7c64fcd81dd\") " pod="kserve/model-serving-api-7b954ffc95-9sjtg"
Sep 18 20:27:27.940234 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.940147    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcjw\" (UniqueName: \"kubernetes.io/projected/148d8816-4714-4a0f-9781-1ab05751e7f4-kube-api-access-hqcjw\") pod \"odh-model-controller-7b966d7984-fsngc\" (UID: \"148d8816-4714-4a0f-9781-1ab05751e7f4\") " pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:27.940234 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.940170    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2cx5\" (UniqueName: \"kubernetes.io/projected/585d62ef-4d14-4ef9-8c75-c7c64fcd81dd-kube-api-access-l2cx5\") pod \"model-serving-api-7b954ffc95-9sjtg\" (UID: \"585d62ef-4d14-4ef9-8c75-c7c64fcd81dd\") " pod="kserve/model-serving-api-7b954ffc95-9sjtg"
Sep 18 20:27:27.940234 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.940190    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/148d8816-4714-4a0f-9781-1ab05751e7f4-cert\") pod \"odh-model-controller-7b966d7984-fsngc\" (UID: \"148d8816-4714-4a0f-9781-1ab05751e7f4\") " pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:27.940234 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:27.940209    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/148d8816-4714-4a0f-9781-1ab05751e7f4-metrics-certs\") pod \"odh-model-controller-7b966d7984-fsngc\" (UID: \"148d8816-4714-4a0f-9781-1ab05751e7f4\") " pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:28.034157 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.034124    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:28.040854 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.040802    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcjw\" (UniqueName: \"kubernetes.io/projected/148d8816-4714-4a0f-9781-1ab05751e7f4-kube-api-access-hqcjw\") pod \"odh-model-controller-7b966d7984-fsngc\" (UID: \"148d8816-4714-4a0f-9781-1ab05751e7f4\") " pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:28.040854 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.040832    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2cx5\" (UniqueName: \"kubernetes.io/projected/585d62ef-4d14-4ef9-8c75-c7c64fcd81dd-kube-api-access-l2cx5\") pod \"model-serving-api-7b954ffc95-9sjtg\" (UID: \"585d62ef-4d14-4ef9-8c75-c7c64fcd81dd\") " pod="kserve/model-serving-api-7b954ffc95-9sjtg"
Sep 18 20:27:28.040854 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.040851    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/148d8816-4714-4a0f-9781-1ab05751e7f4-cert\") pod \"odh-model-controller-7b966d7984-fsngc\" (UID: \"148d8816-4714-4a0f-9781-1ab05751e7f4\") " pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:28.041053 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.040866    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/148d8816-4714-4a0f-9781-1ab05751e7f4-metrics-certs\") pod \"odh-model-controller-7b966d7984-fsngc\" (UID: \"148d8816-4714-4a0f-9781-1ab05751e7f4\") " pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:28.041053 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.040919    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585d62ef-4d14-4ef9-8c75-c7c64fcd81dd-tls-certs\") pod \"model-serving-api-7b954ffc95-9sjtg\" (UID: \"585d62ef-4d14-4ef9-8c75-c7c64fcd81dd\") " pod="kserve/model-serving-api-7b954ffc95-9sjtg"
Sep 18 20:27:28.041053 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:27:28.041035    2557 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 18 20:27:28.041151 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:27:28.041087    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148d8816-4714-4a0f-9781-1ab05751e7f4-metrics-certs podName:148d8816-4714-4a0f-9781-1ab05751e7f4 nodeName:}" failed. No retries permitted until 2026-09-18 20:27:28.541071073 +0000 UTC m=+412.034423630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/148d8816-4714-4a0f-9781-1ab05751e7f4-metrics-certs") pod "odh-model-controller-7b966d7984-fsngc" (UID: "148d8816-4714-4a0f-9781-1ab05751e7f4") : secret "odh-model-controller-metrics-tls" not found
Sep 18 20:27:28.043374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.043355    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585d62ef-4d14-4ef9-8c75-c7c64fcd81dd-tls-certs\") pod \"model-serving-api-7b954ffc95-9sjtg\" (UID: \"585d62ef-4d14-4ef9-8c75-c7c64fcd81dd\") " pod="kserve/model-serving-api-7b954ffc95-9sjtg"
Sep 18 20:27:28.043492 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.043360    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/148d8816-4714-4a0f-9781-1ab05751e7f4-cert\") pod \"odh-model-controller-7b966d7984-fsngc\" (UID: \"148d8816-4714-4a0f-9781-1ab05751e7f4\") " pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:28.053104 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.053082    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcjw\" (UniqueName: \"kubernetes.io/projected/148d8816-4714-4a0f-9781-1ab05751e7f4-kube-api-access-hqcjw\") pod \"odh-model-controller-7b966d7984-fsngc\" (UID: \"148d8816-4714-4a0f-9781-1ab05751e7f4\") " pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:28.056120 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.056099    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2cx5\" (UniqueName: \"kubernetes.io/projected/585d62ef-4d14-4ef9-8c75-c7c64fcd81dd-kube-api-access-l2cx5\") pod \"model-serving-api-7b954ffc95-9sjtg\" (UID: \"585d62ef-4d14-4ef9-8c75-c7c64fcd81dd\") " pod="kserve/model-serving-api-7b954ffc95-9sjtg"
Sep 18 20:27:28.134263 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.134238    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:28.149410 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.149389    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-9sjtg"
Sep 18 20:27:28.423577 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.423432    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-9sjtg"]
Sep 18 20:27:28.426286 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:27:28.426260    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585d62ef_4d14_4ef9_8c75_c7c64fcd81dd.slice/crio-98da7df16e6f067f64c3416720be2099fc8e3017a663e1caee6c8f0aec09253a WatchSource:0}: Error finding container 98da7df16e6f067f64c3416720be2099fc8e3017a663e1caee6c8f0aec09253a: Status 404 returned error can't find the container with id 98da7df16e6f067f64c3416720be2099fc8e3017a663e1caee6c8f0aec09253a
Sep 18 20:27:28.544624 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.544590    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/148d8816-4714-4a0f-9781-1ab05751e7f4-metrics-certs\") pod \"odh-model-controller-7b966d7984-fsngc\" (UID: \"148d8816-4714-4a0f-9781-1ab05751e7f4\") " pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:28.546883 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.546857    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/148d8816-4714-4a0f-9781-1ab05751e7f4-metrics-certs\") pod \"odh-model-controller-7b966d7984-fsngc\" (UID: \"148d8816-4714-4a0f-9781-1ab05751e7f4\") " pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:28.639584 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.639557    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-9sjtg" event={"ID":"585d62ef-4d14-4ef9-8c75-c7c64fcd81dd","Type":"ContainerStarted","Data":"98da7df16e6f067f64c3416720be2099fc8e3017a663e1caee6c8f0aec09253a"}
Sep 18 20:27:28.774276 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:28.774246    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:29.035690 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:29.035611    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:29.050213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:29.050183    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-fsngc"]
Sep 18 20:27:29.055046 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:27:29.055017    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148d8816_4714_4a0f_9781_1ab05751e7f4.slice/crio-a6d042a5de009a5b03ecf607c6e97ac918e7617bf6a5638148ff1ddf009e8933 WatchSource:0}: Error finding container a6d042a5de009a5b03ecf607c6e97ac918e7617bf6a5638148ff1ddf009e8933: Status 404 returned error can't find the container with id a6d042a5de009a5b03ecf607c6e97ac918e7617bf6a5638148ff1ddf009e8933
Sep 18 20:27:29.132372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:29.132333    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:29.645335 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:29.645290    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-fsngc" event={"ID":"148d8816-4714-4a0f-9781-1ab05751e7f4","Type":"ContainerStarted","Data":"a6d042a5de009a5b03ecf607c6e97ac918e7617bf6a5638148ff1ddf009e8933"}
Sep 18 20:27:30.652007 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:30.651966    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-9sjtg" event={"ID":"585d62ef-4d14-4ef9-8c75-c7c64fcd81dd","Type":"ContainerStarted","Data":"327e0a472817e5f033ee57d7163ceb321227b2b9f71c8cb8a55318d07708b08f"}
Sep 18 20:27:30.652493 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:30.652066    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-9sjtg"
Sep 18 20:27:30.669287 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:30.669228    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-9sjtg" podStartSLOduration=2.199245198 podStartE2EDuration="3.669207928s" totalImagesPullingTime="1.46996273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:27:30.667321841 +0000 UTC m=+414.160674424" watchObservedRunningTime="2026-09-18 20:27:30.669207928 +0000 UTC m=+414.162560508"
Sep 18 20:27:32.664977 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:32.664937    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-fsngc" event={"ID":"148d8816-4714-4a0f-9781-1ab05751e7f4","Type":"ContainerStarted","Data":"9b4ab171bdd64e6f93acbcbd266407955c35ea3888190e505c2e35aac631226c"}
Sep 18 20:27:32.665440 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:32.665074    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:27:32.683334 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:32.683284    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-fsngc" podStartSLOduration=2.615459201 podStartE2EDuration="5.683268066s" totalImagesPullingTime="3.067808865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:27:32.681522075 +0000 UTC m=+416.174874651" watchObservedRunningTime="2026-09-18 20:27:32.683268066 +0000 UTC m=+416.176620644"
Sep 18 20:27:41.661516 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:41.661449    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-9sjtg"
Sep 18 20:27:43.671801 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:27:43.671759    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-fsngc"
Sep 18 20:28:11.015411 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:28:11.015372    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:38.285788 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:28:38.285739    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:03.767168 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:03.767133    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:03.866519 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:03.866483    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:15.656806 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.656723    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:15.854988 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.854952    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:15.932243 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.932208    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:15.933428 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.933405    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"]
Sep 18 20:29:15.936037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.936018    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:15.938424 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.938390    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:29:15.938570 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.938427    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:29:15.938570 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.938397    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:29:15.938570 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.938468    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:29:15.938570 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.938481    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 20:29:15.938570 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.938512    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 20:29:15.938938 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.938697    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ttgfx\""
Sep 18 20:29:15.947875 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.947852    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"]
Sep 18 20:29:15.955532 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.955501    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5de5c92-6932-4f74-874c-00f4dd12d269-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:15.955737 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.955548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8km2\" (UniqueName: \"kubernetes.io/projected/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-api-access-r8km2\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:15.955737 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.955578    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5de5c92-6932-4f74-874c-00f4dd12d269-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:15.955737 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.955646    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:15.955951 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:15.955799    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.023103 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.023058    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:16.056328 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.056292    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.056528 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.056349    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5de5c92-6932-4f74-874c-00f4dd12d269-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.056528 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.056480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8km2\" (UniqueName: \"kubernetes.io/projected/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-api-access-r8km2\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.056670 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.056529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5de5c92-6932-4f74-874c-00f4dd12d269-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.056670 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.056552    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.056943 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.056918    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5de5c92-6932-4f74-874c-00f4dd12d269-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.057146 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.057125    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.057204 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.057175    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.058841 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.058820    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5de5c92-6932-4f74-874c-00f4dd12d269-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.064026 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.064005    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8km2\" (UniqueName: \"kubernetes.io/projected/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-api-access-r8km2\") pod \"isvc-canary-rollback-predictor-7f7b644c9-gr84x\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.142102 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.142065    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:16.248364 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.248263    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:16.530854 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:16.530826    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"]
Sep 18 20:29:16.533723 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:29:16.533686    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5de5c92_6932_4f74_874c_00f4dd12d269.slice/crio-6227f7808a186780bb943a90476b4abc0ff3da164b38b72b69d5f85d52be11fa WatchSource:0}: Error finding container 6227f7808a186780bb943a90476b4abc0ff3da164b38b72b69d5f85d52be11fa: Status 404 returned error can't find the container with id 6227f7808a186780bb943a90476b4abc0ff3da164b38b72b69d5f85d52be11fa
Sep 18 20:29:17.132520 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:17.132487    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" event={"ID":"a5de5c92-6932-4f74-874c-00f4dd12d269","Type":"ContainerStarted","Data":"6227f7808a186780bb943a90476b4abc0ff3da164b38b72b69d5f85d52be11fa"}
Sep 18 20:29:20.149619 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:20.149581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" event={"ID":"a5de5c92-6932-4f74-874c-00f4dd12d269","Type":"ContainerStarted","Data":"828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5"}
Sep 18 20:29:20.758375 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:20.758340    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:24.170817 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:24.170779    2557 generic.go:356] "Generic (PLEG): container finished" podID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerID="828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5" exitCode=0
Sep 18 20:29:24.171252 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:24.170852    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" event={"ID":"a5de5c92-6932-4f74-874c-00f4dd12d269","Type":"ContainerDied","Data":"828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5"}
Sep 18 20:29:28.922046 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:28.922004    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:38.258937 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:38.258886    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" event={"ID":"a5de5c92-6932-4f74-874c-00f4dd12d269","Type":"ContainerStarted","Data":"fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f"}
Sep 18 20:29:40.271708 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:40.271671    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" event={"ID":"a5de5c92-6932-4f74-874c-00f4dd12d269","Type":"ContainerStarted","Data":"e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b"}
Sep 18 20:29:40.272137 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:40.271742    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:40.272137 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:40.271772    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:40.273496 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:40.273442    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:29:40.291113 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:40.291052    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podStartSLOduration=2.404791678 podStartE2EDuration="25.291031739s" totalImagesPullingTime="18.945664129s" totalInitContainerRuntime="3.940575932s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:40.289588375 +0000 UTC m=+543.782940954" watchObservedRunningTime="2026-09-18 20:29:40.291031739 +0000 UTC m=+543.784384318"
Sep 18 20:29:41.277875 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:41.277830    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:29:46.282956 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:46.282925    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:29:46.283420 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:46.283393    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:29:56.283911 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:29:56.283871    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:30:06.284059 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:30:06.284023    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:30:16.284026 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:30:16.283986    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:30:26.283901 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:30:26.283860    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:30:36.283887 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:30:36.283844    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:30:38.640314 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:30:38.640285    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:30:38.641056 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:30:38.641032    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:30:46.284273 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:30:46.284227    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:30:47.717900 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:30:47.717856    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:30:57.719388 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:30:57.719359    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:31:05.686249 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:05.686208    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:11.090447 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.090403    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:11.188692 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.188657    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:11.222910 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.222874    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"]
Sep 18 20:31:11.226442 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.226416    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.228508 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.228482    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-12fdc-kube-rbac-proxy-sar-config\""
Sep 18 20:31:11.228626 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.228559    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-12fdc-predictor-serving-cert\""
Sep 18 20:31:11.241750 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.241720    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"]
Sep 18 20:31:11.312704 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.312666    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:11.341736 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.341663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40eccc2-afa9-41d3-b828-d91a8eb096be-proxy-tls\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.341736 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.341699    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncdht\" (UniqueName: \"kubernetes.io/projected/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-api-access-ncdht\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.341906 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.341747    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.341906 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.341779    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-cabundle-cert\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.341906 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.341833    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f40eccc2-afa9-41d3-b828-d91a8eb096be-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.432404 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.432363    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:11.442516 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.442495    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40eccc2-afa9-41d3-b828-d91a8eb096be-proxy-tls\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.442630 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.442528    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncdht\" (UniqueName: \"kubernetes.io/projected/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-api-access-ncdht\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.442630 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.442559    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.442630 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.442575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-cabundle-cert\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.442630 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.442599    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f40eccc2-afa9-41d3-b828-d91a8eb096be-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.442837 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:31:11.442664    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-serving-cert: secret "isvc-sklearn-graph-raw-12fdc-predictor-serving-cert" not found
Sep 18 20:31:11.442837 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:31:11.442729    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f40eccc2-afa9-41d3-b828-d91a8eb096be-proxy-tls podName:f40eccc2-afa9-41d3-b828-d91a8eb096be nodeName:}" failed. No retries permitted until 2026-09-18 20:31:11.942709236 +0000 UTC m=+635.436061811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f40eccc2-afa9-41d3-b828-d91a8eb096be-proxy-tls") pod "isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" (UID: "f40eccc2-afa9-41d3-b828-d91a8eb096be") : secret "isvc-sklearn-graph-raw-12fdc-predictor-serving-cert" not found
Sep 18 20:31:11.443004 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.442986    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f40eccc2-afa9-41d3-b828-d91a8eb096be-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.443292 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.443274    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.443329 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.443284    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-cabundle-cert\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.453822 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.453797    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncdht\" (UniqueName: \"kubernetes.io/projected/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-api-access-ncdht\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.559369 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.559339    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:11.945872 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.945835    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40eccc2-afa9-41d3-b828-d91a8eb096be-proxy-tls\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:11.948329 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:11.948306    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40eccc2-afa9-41d3-b828-d91a8eb096be-proxy-tls\") pod \"isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:12.142138 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:12.142100    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:12.413545 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:12.413520    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"]
Sep 18 20:31:12.415355 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:31:12.415325    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40eccc2_afa9_41d3_b828_d91a8eb096be.slice/crio-6493df3ef5d32a466a96e6a2d1802fea11bbe96fe4af6fa34038043f07bfd7b9 WatchSource:0}: Error finding container 6493df3ef5d32a466a96e6a2d1802fea11bbe96fe4af6fa34038043f07bfd7b9: Status 404 returned error can't find the container with id 6493df3ef5d32a466a96e6a2d1802fea11bbe96fe4af6fa34038043f07bfd7b9
Sep 18 20:31:12.700224 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:12.700188    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" event={"ID":"f40eccc2-afa9-41d3-b828-d91a8eb096be","Type":"ContainerStarted","Data":"bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55"}
Sep 18 20:31:12.700412 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:12.700247    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" event={"ID":"f40eccc2-afa9-41d3-b828-d91a8eb096be","Type":"ContainerStarted","Data":"6493df3ef5d32a466a96e6a2d1802fea11bbe96fe4af6fa34038043f07bfd7b9"}
Sep 18 20:31:16.719083 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:16.719054    2557 generic.go:356] "Generic (PLEG): container finished" podID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerID="bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55" exitCode=0
Sep 18 20:31:16.722873 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:16.722849    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" event={"ID":"f40eccc2-afa9-41d3-b828-d91a8eb096be","Type":"ContainerDied","Data":"bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55"}
Sep 18 20:31:17.726296 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:17.726263    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" event={"ID":"f40eccc2-afa9-41d3-b828-d91a8eb096be","Type":"ContainerStarted","Data":"c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d"}
Sep 18 20:31:17.726729 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:17.726352    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" event={"ID":"f40eccc2-afa9-41d3-b828-d91a8eb096be","Type":"ContainerStarted","Data":"109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0"}
Sep 18 20:31:17.726729 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:17.726648    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:17.746588 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:17.746548    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podStartSLOduration=2.841397411 podStartE2EDuration="6.746532669s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.905135258s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:31:17.744223348 +0000 UTC m=+641.237575926" watchObservedRunningTime="2026-09-18 20:31:17.746532669 +0000 UTC m=+641.239885250"
Sep 18 20:31:18.731138 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:18.731104    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:18.732368 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:18.732334    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:31:19.735932 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:19.735891    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:31:24.740175 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:24.740148    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:31:24.740611 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:24.740586    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:31:29.909907 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:29.909862    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:34.741103 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:34.741063    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:31:44.740647 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:44.740610    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:31:51.545027 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:51.544984    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:54.741292 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:31:54.741248    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:32:02.883244 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:02.883200    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:04.740576 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:04.740532    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:32:14.741106 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:14.741064    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:32:24.741374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:24.741287    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:32:34.741432 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:34.741401    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:32:35.771668 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:35.771634    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"]
Sep 18 20:32:35.772126 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:35.771899    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" containerID="cri-o://fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f" gracePeriod=30
Sep 18 20:32:35.772126 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:35.771949    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kube-rbac-proxy" containerID="cri-o://e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b" gracePeriod=30
Sep 18 20:32:35.801951 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:35.801908    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:36.091803 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:36.091720    2557 generic.go:356] "Generic (PLEG): container finished" podID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerID="e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b" exitCode=2
Sep 18 20:32:36.091803 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:36.091770    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" event={"ID":"a5de5c92-6932-4f74-874c-00f4dd12d269","Type":"ContainerDied","Data":"e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b"}
Sep 18 20:32:36.278860 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:36.278821    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 18 20:32:37.717760 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:37.717713    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:32:40.511862 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.511839    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:32:40.693505 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.693481    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-cabundle-cert\") pod \"a5de5c92-6932-4f74-874c-00f4dd12d269\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") "
Sep 18 20:32:40.693660 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.693543    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-rbac-proxy-sar-config\") pod \"a5de5c92-6932-4f74-874c-00f4dd12d269\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") "
Sep 18 20:32:40.693660 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.693649    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5de5c92-6932-4f74-874c-00f4dd12d269-kserve-provision-location\") pod \"a5de5c92-6932-4f74-874c-00f4dd12d269\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") "
Sep 18 20:32:40.693738 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.693710    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8km2\" (UniqueName: \"kubernetes.io/projected/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-api-access-r8km2\") pod \"a5de5c92-6932-4f74-874c-00f4dd12d269\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") "
Sep 18 20:32:40.693775 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.693758    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5de5c92-6932-4f74-874c-00f4dd12d269-proxy-tls\") pod \"a5de5c92-6932-4f74-874c-00f4dd12d269\" (UID: \"a5de5c92-6932-4f74-874c-00f4dd12d269\") "
Sep 18 20:32:40.693904 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.693881    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5de5c92-6932-4f74-874c-00f4dd12d269-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5de5c92-6932-4f74-874c-00f4dd12d269" (UID: "a5de5c92-6932-4f74-874c-00f4dd12d269"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:40.693904 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.693889    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5de5c92-6932-4f74-874c-00f4dd12d269" (UID: "a5de5c92-6932-4f74-874c-00f4dd12d269"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:40.693904 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.693898    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5de5c92-6932-4f74-874c-00f4dd12d269" (UID: "a5de5c92-6932-4f74-874c-00f4dd12d269"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:40.694085 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.694051    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-cabundle-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:32:40.694085 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.694071    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:32:40.694085 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.694085    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5de5c92-6932-4f74-874c-00f4dd12d269-kserve-provision-location\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:32:40.695905 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.695883    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5de5c92-6932-4f74-874c-00f4dd12d269-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5de5c92-6932-4f74-874c-00f4dd12d269" (UID: "a5de5c92-6932-4f74-874c-00f4dd12d269"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:32:40.695974 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.695908    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-api-access-r8km2" (OuterVolumeSpecName: "kube-api-access-r8km2") pod "a5de5c92-6932-4f74-874c-00f4dd12d269" (UID: "a5de5c92-6932-4f74-874c-00f4dd12d269"). InnerVolumeSpecName "kube-api-access-r8km2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:40.794874 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.794842    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8km2\" (UniqueName: \"kubernetes.io/projected/a5de5c92-6932-4f74-874c-00f4dd12d269-kube-api-access-r8km2\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:32:40.794874 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.794875    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5de5c92-6932-4f74-874c-00f4dd12d269-proxy-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:32:40.988053 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:40.987958    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:41.064463 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.064414    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"]
Sep 18 20:32:41.064846 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.064832    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="storage-initializer"
Sep 18 20:32:41.064891 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.064848    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="storage-initializer"
Sep 18 20:32:41.064891 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.064863    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kube-rbac-proxy"
Sep 18 20:32:41.064891 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.064876    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kube-rbac-proxy"
Sep 18 20:32:41.064891 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.064887    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container"
Sep 18 20:32:41.065015 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.064895    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container"
Sep 18 20:32:41.065015 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.064977    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kserve-container"
Sep 18 20:32:41.065015 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.064986    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerName="kube-rbac-proxy"
Sep 18 20:32:41.068574 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.068555    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.070653 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.070628    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 20:32:41.070807 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.070688    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 20:32:41.077913 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.077889    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"]
Sep 18 20:32:41.083076 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.083037    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:41.118153 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.118118    2557 generic.go:356] "Generic (PLEG): container finished" podID="a5de5c92-6932-4f74-874c-00f4dd12d269" containerID="fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f" exitCode=0
Sep 18 20:32:41.118305 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.118170    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"
Sep 18 20:32:41.118305 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.118170    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" event={"ID":"a5de5c92-6932-4f74-874c-00f4dd12d269","Type":"ContainerDied","Data":"fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f"}
Sep 18 20:32:41.118305 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.118298    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x" event={"ID":"a5de5c92-6932-4f74-874c-00f4dd12d269","Type":"ContainerDied","Data":"6227f7808a186780bb943a90476b4abc0ff3da164b38b72b69d5f85d52be11fa"}
Sep 18 20:32:41.118414 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.118328    2557 scope.go:117] "RemoveContainer" containerID="e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b"
Sep 18 20:32:41.126817 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.126798    2557 scope.go:117] "RemoveContainer" containerID="fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f"
Sep 18 20:32:41.134520 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.134495    2557 scope.go:117] "RemoveContainer" containerID="828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5"
Sep 18 20:32:41.141688 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.141671    2557 scope.go:117] "RemoveContainer" containerID="e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b"
Sep 18 20:32:41.141969 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:32:41.141948    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b\": container with ID starting with e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b not found: ID does not exist" containerID="e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b"
Sep 18 20:32:41.142023 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.141977    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b"} err="failed to get container status \"e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b\": rpc error: code = NotFound desc = could not find container \"e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b\": container with ID starting with e94ed2e4f1e294e9b2c2bddaab8c8aa069c2d4976b9deca908ac3102993ce26b not found: ID does not exist"
Sep 18 20:32:41.142023 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.141995    2557 scope.go:117] "RemoveContainer" containerID="fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f"
Sep 18 20:32:41.142412 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:32:41.142370    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f\": container with ID starting with fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f not found: ID does not exist" containerID="fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f"
Sep 18 20:32:41.142412 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.142400    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f"} err="failed to get container status \"fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f\": rpc error: code = NotFound desc = could not find container \"fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f\": container with ID starting with fe77333526191777cdf6b21ec56d8e504ec1c1b1867c4332cf66b884de72d63f not found: ID does not exist"
Sep 18 20:32:41.142657 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.142420    2557 scope.go:117] "RemoveContainer" containerID="828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5"
Sep 18 20:32:41.142997 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:32:41.142977    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5\": container with ID starting with 828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5 not found: ID does not exist" containerID="828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5"
Sep 18 20:32:41.143071 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.143005    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5"} err="failed to get container status \"828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5\": rpc error: code = NotFound desc = could not find container \"828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5\": container with ID starting with 828305d1d9bf9d3f6554ca1122f9f484a5e8d5c8c95578e8ef6702aa3f5b49a5 not found: ID does not exist"
Sep 18 20:32:41.150800 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.150778    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"]
Sep 18 20:32:41.159109 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.159087    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-gr84x"]
Sep 18 20:32:41.168356 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.168332    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:41.199122 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.199095    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkns9\" (UniqueName: \"kubernetes.io/projected/848c20cf-6352-45fc-84ec-209370583788-kube-api-access-tkns9\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.199262 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.199128    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/848c20cf-6352-45fc-84ec-209370583788-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.199262 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.199149    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.199262 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.199165    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/848c20cf-6352-45fc-84ec-209370583788-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.199363 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.199269    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.300161 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.300080    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkns9\" (UniqueName: \"kubernetes.io/projected/848c20cf-6352-45fc-84ec-209370583788-kube-api-access-tkns9\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.300161 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.300114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/848c20cf-6352-45fc-84ec-209370583788-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.300161 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.300133    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.300161 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.300148    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/848c20cf-6352-45fc-84ec-209370583788-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.300526 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.300198    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.300595 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.300558    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/848c20cf-6352-45fc-84ec-209370583788-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.300943 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.300916    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.301064 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.300943    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.303031 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.303007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/848c20cf-6352-45fc-84ec-209370583788-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.308464 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.308419    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkns9\" (UniqueName: \"kubernetes.io/projected/848c20cf-6352-45fc-84ec-209370583788-kube-api-access-tkns9\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6w89x\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.328523 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.328487    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:41.380384 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.380339    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:41.656365 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:41.656312    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"]
Sep 18 20:32:41.658802 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:32:41.658770    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848c20cf_6352_45fc_84ec_209370583788.slice/crio-660831c21302bf8b7b06a9cda1f9bb479cd3f759409217250ba0fd170e26cace WatchSource:0}: Error finding container 660831c21302bf8b7b06a9cda1f9bb479cd3f759409217250ba0fd170e26cace: Status 404 returned error can't find the container with id 660831c21302bf8b7b06a9cda1f9bb479cd3f759409217250ba0fd170e26cace
Sep 18 20:32:42.124342 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:42.124295    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" event={"ID":"848c20cf-6352-45fc-84ec-209370583788","Type":"ContainerStarted","Data":"1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9"}
Sep 18 20:32:42.124569 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:42.124413    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" event={"ID":"848c20cf-6352-45fc-84ec-209370583788","Type":"ContainerStarted","Data":"660831c21302bf8b7b06a9cda1f9bb479cd3f759409217250ba0fd170e26cace"}
Sep 18 20:32:42.723728 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:42.723695    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5de5c92-6932-4f74-874c-00f4dd12d269" path="/var/lib/kubelet/pods/a5de5c92-6932-4f74-874c-00f4dd12d269/volumes"
Sep 18 20:32:46.146274 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:46.146239    2557 generic.go:356] "Generic (PLEG): container finished" podID="848c20cf-6352-45fc-84ec-209370583788" containerID="1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9" exitCode=0
Sep 18 20:32:46.146660 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:46.146311    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" event={"ID":"848c20cf-6352-45fc-84ec-209370583788","Type":"ContainerDied","Data":"1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9"}
Sep 18 20:32:47.155113 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:47.155076    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" event={"ID":"848c20cf-6352-45fc-84ec-209370583788","Type":"ContainerStarted","Data":"4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add"}
Sep 18 20:32:47.155587 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:47.155136    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" event={"ID":"848c20cf-6352-45fc-84ec-209370583788","Type":"ContainerStarted","Data":"b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378"}
Sep 18 20:32:47.155587 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:47.155315    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:47.177216 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:47.177165    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podStartSLOduration=2.335070924 podStartE2EDuration="6.177149605s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.842078681s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:47.17592098 +0000 UTC m=+730.669273574" watchObservedRunningTime="2026-09-18 20:32:47.177149605 +0000 UTC m=+730.670502186"
Sep 18 20:32:48.159995 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:48.159961    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:48.161347 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:48.161307    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:32:49.165212 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:49.165159    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:32:51.263786 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:51.263750    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:51.584148 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:51.584058    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:54.169800 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:54.169773    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:32:54.170271 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:32:54.170244    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:33:01.292267 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:01.292232    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:01.374718 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:01.374685    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"]
Sep 18 20:33:01.374948 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:01.374923    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" containerID="cri-o://109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0" gracePeriod=30
Sep 18 20:33:01.375004 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:01.374953    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kube-rbac-proxy" containerID="cri-o://c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d" gracePeriod=30
Sep 18 20:33:01.408730 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:01.408686    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:01.510228 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:01.510176    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:01.585183 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:01.585084    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:01.651908 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:01.651868    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:01.879531 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:01.879428    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:02.007871 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:02.007836    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:02.229589 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:02.229558    2557 generic.go:356] "Generic (PLEG): container finished" podID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerID="c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d" exitCode=2
Sep 18 20:33:02.229768 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:02.229635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" event={"ID":"f40eccc2-afa9-41d3-b828-d91a8eb096be","Type":"ContainerDied","Data":"c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d"}
Sep 18 20:33:04.170286 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:04.170248    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:33:04.736080 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:04.736042    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 18 20:33:04.741386 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:04.741364    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:33:06.515798 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.515767    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:33:06.620864 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.620776    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40eccc2-afa9-41d3-b828-d91a8eb096be-proxy-tls\") pod \"f40eccc2-afa9-41d3-b828-d91a8eb096be\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") "
Sep 18 20:33:06.620864 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.620813    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncdht\" (UniqueName: \"kubernetes.io/projected/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-api-access-ncdht\") pod \"f40eccc2-afa9-41d3-b828-d91a8eb096be\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") "
Sep 18 20:33:06.620864 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.620835    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-rbac-proxy-sar-config\") pod \"f40eccc2-afa9-41d3-b828-d91a8eb096be\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") "
Sep 18 20:33:06.620864 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.620856    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f40eccc2-afa9-41d3-b828-d91a8eb096be-kserve-provision-location\") pod \"f40eccc2-afa9-41d3-b828-d91a8eb096be\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") "
Sep 18 20:33:06.621189 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.620918    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-cabundle-cert\") pod \"f40eccc2-afa9-41d3-b828-d91a8eb096be\" (UID: \"f40eccc2-afa9-41d3-b828-d91a8eb096be\") "
Sep 18 20:33:06.621247 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.621213    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40eccc2-afa9-41d3-b828-d91a8eb096be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f40eccc2-afa9-41d3-b828-d91a8eb096be" (UID: "f40eccc2-afa9-41d3-b828-d91a8eb096be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:33:06.621285 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.621243    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f40eccc2-afa9-41d3-b828-d91a8eb096be" (UID: "f40eccc2-afa9-41d3-b828-d91a8eb096be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:33:06.621334 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.621308    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f40eccc2-afa9-41d3-b828-d91a8eb096be" (UID: "f40eccc2-afa9-41d3-b828-d91a8eb096be"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:33:06.622961 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.622937    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-api-access-ncdht" (OuterVolumeSpecName: "kube-api-access-ncdht") pod "f40eccc2-afa9-41d3-b828-d91a8eb096be" (UID: "f40eccc2-afa9-41d3-b828-d91a8eb096be"). InnerVolumeSpecName "kube-api-access-ncdht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:33:06.623045 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.622940    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40eccc2-afa9-41d3-b828-d91a8eb096be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f40eccc2-afa9-41d3-b828-d91a8eb096be" (UID: "f40eccc2-afa9-41d3-b828-d91a8eb096be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:33:06.721982 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.721950    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-cabundle-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:33:06.722117 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.721992    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40eccc2-afa9-41d3-b828-d91a8eb096be-proxy-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:33:06.722117 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.722002    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncdht\" (UniqueName: \"kubernetes.io/projected/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-api-access-ncdht\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:33:06.722117 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.722012    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40eccc2-afa9-41d3-b828-d91a8eb096be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:33:06.722117 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:06.722020    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f40eccc2-afa9-41d3-b828-d91a8eb096be-kserve-provision-location\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:33:07.257602 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.257565    2557 generic.go:356] "Generic (PLEG): container finished" podID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerID="109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0" exitCode=0
Sep 18 20:33:07.257786 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.257623    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"
Sep 18 20:33:07.257786 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.257648    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" event={"ID":"f40eccc2-afa9-41d3-b828-d91a8eb096be","Type":"ContainerDied","Data":"109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0"}
Sep 18 20:33:07.257786 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.257713    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg" event={"ID":"f40eccc2-afa9-41d3-b828-d91a8eb096be","Type":"ContainerDied","Data":"6493df3ef5d32a466a96e6a2d1802fea11bbe96fe4af6fa34038043f07bfd7b9"}
Sep 18 20:33:07.257786 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.257739    2557 scope.go:117] "RemoveContainer" containerID="c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d"
Sep 18 20:33:07.266480 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.266442    2557 scope.go:117] "RemoveContainer" containerID="109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0"
Sep 18 20:33:07.273790 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.273766    2557 scope.go:117] "RemoveContainer" containerID="bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55"
Sep 18 20:33:07.275690 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.275669    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"]
Sep 18 20:33:07.279218 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.279199    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-12fdc-predictor-58cb5fb5d-89xcg"]
Sep 18 20:33:07.281987 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.281972    2557 scope.go:117] "RemoveContainer" containerID="c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d"
Sep 18 20:33:07.282237 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:33:07.282220    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d\": container with ID starting with c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d not found: ID does not exist" containerID="c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d"
Sep 18 20:33:07.282294 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.282247    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d"} err="failed to get container status \"c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d\": rpc error: code = NotFound desc = could not find container \"c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d\": container with ID starting with c750dbeb42509efd51a9fff412fdacb6fee26fc21c885865bcf74b551842b44d not found: ID does not exist"
Sep 18 20:33:07.282294 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.282262    2557 scope.go:117] "RemoveContainer" containerID="109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0"
Sep 18 20:33:07.282535 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:33:07.282517    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0\": container with ID starting with 109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0 not found: ID does not exist" containerID="109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0"
Sep 18 20:33:07.282580 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.282543    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0"} err="failed to get container status \"109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0\": rpc error: code = NotFound desc = could not find container \"109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0\": container with ID starting with 109d3fb67ff1f32ad85052a33ae4404d59b1fc84ea2945147d91c56c9f4f53e0 not found: ID does not exist"
Sep 18 20:33:07.282580 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.282559    2557 scope.go:117] "RemoveContainer" containerID="bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55"
Sep 18 20:33:07.282804 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:33:07.282784    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55\": container with ID starting with bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55 not found: ID does not exist" containerID="bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55"
Sep 18 20:33:07.282855 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:07.282811    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55"} err="failed to get container status \"bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55\": rpc error: code = NotFound desc = could not find container \"bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55\": container with ID starting with bf0ec7c9bdd1ddcf7b41492f022c127d1e6be586a45ba2ab4760c0e501863d55 not found: ID does not exist"
Sep 18 20:33:08.723390 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:08.723357    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" path="/var/lib/kubelet/pods/f40eccc2-afa9-41d3-b828-d91a8eb096be/volumes"
Sep 18 20:33:14.170559 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:14.170515    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:33:16.908864 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:16.908832    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:16.974966 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:16.974932    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:24.171098 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:24.171060    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:33:34.170232 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:34.170191    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:33:44.171142 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:44.171100    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:33:54.170772 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:33:54.170730    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:34:04.171626 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:04.171597    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:34:15.975204 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:15.975171    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:16.012883 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:16.012850    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"]
Sep 18 20:34:16.013099 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:16.013077    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container" containerID="cri-o://b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378" gracePeriod=30
Sep 18 20:34:16.013188 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:16.013127    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kube-rbac-proxy" containerID="cri-o://4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add" gracePeriod=30
Sep 18 20:34:16.050770 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:16.050744    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:16.584054 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:16.584025    2557 generic.go:356] "Generic (PLEG): container finished" podID="848c20cf-6352-45fc-84ec-209370583788" containerID="4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add" exitCode=2
Sep 18 20:34:16.584213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:16.584099    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" event={"ID":"848c20cf-6352-45fc-84ec-209370583788","Type":"ContainerDied","Data":"4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add"}
Sep 18 20:34:19.166194 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:19.166148    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 18 20:34:20.762927 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.762901    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:34:20.863666 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.863587    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/848c20cf-6352-45fc-84ec-209370583788-kserve-provision-location\") pod \"848c20cf-6352-45fc-84ec-209370583788\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") "
Sep 18 20:34:20.863666 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.863635    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-cabundle-cert\") pod \"848c20cf-6352-45fc-84ec-209370583788\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") "
Sep 18 20:34:20.863666 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.863665    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/848c20cf-6352-45fc-84ec-209370583788-proxy-tls\") pod \"848c20cf-6352-45fc-84ec-209370583788\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") "
Sep 18 20:34:20.863933 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.863722    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkns9\" (UniqueName: \"kubernetes.io/projected/848c20cf-6352-45fc-84ec-209370583788-kube-api-access-tkns9\") pod \"848c20cf-6352-45fc-84ec-209370583788\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") "
Sep 18 20:34:20.863933 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.863773    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-kube-rbac-proxy-sar-config\") pod \"848c20cf-6352-45fc-84ec-209370583788\" (UID: \"848c20cf-6352-45fc-84ec-209370583788\") "
Sep 18 20:34:20.864238 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.863926    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848c20cf-6352-45fc-84ec-209370583788-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "848c20cf-6352-45fc-84ec-209370583788" (UID: "848c20cf-6352-45fc-84ec-209370583788"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:34:20.864238 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.863978    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "848c20cf-6352-45fc-84ec-209370583788" (UID: "848c20cf-6352-45fc-84ec-209370583788"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:34:20.864238 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.864091    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/848c20cf-6352-45fc-84ec-209370583788-kserve-provision-location\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:34:20.864238 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.864152    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "848c20cf-6352-45fc-84ec-209370583788" (UID: "848c20cf-6352-45fc-84ec-209370583788"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:34:20.865819 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.865800    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848c20cf-6352-45fc-84ec-209370583788-kube-api-access-tkns9" (OuterVolumeSpecName: "kube-api-access-tkns9") pod "848c20cf-6352-45fc-84ec-209370583788" (UID: "848c20cf-6352-45fc-84ec-209370583788"). InnerVolumeSpecName "kube-api-access-tkns9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:34:20.865819 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.865796    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/848c20cf-6352-45fc-84ec-209370583788-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "848c20cf-6352-45fc-84ec-209370583788" (UID: "848c20cf-6352-45fc-84ec-209370583788"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:34:20.964511 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.964475    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:34:20.964511 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.964509    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/848c20cf-6352-45fc-84ec-209370583788-cabundle-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:34:20.964511 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.964519    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/848c20cf-6352-45fc-84ec-209370583788-proxy-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:34:20.964722 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:20.964528    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkns9\" (UniqueName: \"kubernetes.io/projected/848c20cf-6352-45fc-84ec-209370583788-kube-api-access-tkns9\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:34:21.608296 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.608258    2557 generic.go:356] "Generic (PLEG): container finished" podID="848c20cf-6352-45fc-84ec-209370583788" containerID="b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378" exitCode=0
Sep 18 20:34:21.608483 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.608311    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"
Sep 18 20:34:21.608483 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.608350    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" event={"ID":"848c20cf-6352-45fc-84ec-209370583788","Type":"ContainerDied","Data":"b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378"}
Sep 18 20:34:21.608483 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.608415    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x" event={"ID":"848c20cf-6352-45fc-84ec-209370583788","Type":"ContainerDied","Data":"660831c21302bf8b7b06a9cda1f9bb479cd3f759409217250ba0fd170e26cace"}
Sep 18 20:34:21.608483 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.608443    2557 scope.go:117] "RemoveContainer" containerID="4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add"
Sep 18 20:34:21.617213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.617195    2557 scope.go:117] "RemoveContainer" containerID="b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378"
Sep 18 20:34:21.624968 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.624952    2557 scope.go:117] "RemoveContainer" containerID="1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9"
Sep 18 20:34:21.632052 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.632032    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"]
Sep 18 20:34:21.632427 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.632271    2557 scope.go:117] "RemoveContainer" containerID="4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add"
Sep 18 20:34:21.632534 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:34:21.632515    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add\": container with ID starting with 4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add not found: ID does not exist" containerID="4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add"
Sep 18 20:34:21.632591 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.632544    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add"} err="failed to get container status \"4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add\": rpc error: code = NotFound desc = could not find container \"4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add\": container with ID starting with 4c60413f793f1f55d9a66f74ac29232bed3cd94ba22b7007179b907c3ac29add not found: ID does not exist"
Sep 18 20:34:21.632591 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.632562    2557 scope.go:117] "RemoveContainer" containerID="b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378"
Sep 18 20:34:21.632808 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:34:21.632791    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378\": container with ID starting with b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378 not found: ID does not exist" containerID="b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378"
Sep 18 20:34:21.632848 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.632814    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378"} err="failed to get container status \"b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378\": rpc error: code = NotFound desc = could not find container \"b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378\": container with ID starting with b611c992a16180cba834b391443bab613ad6dab3e4fc935da72a841ec5bba378 not found: ID does not exist"
Sep 18 20:34:21.632848 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.632828    2557 scope.go:117] "RemoveContainer" containerID="1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9"
Sep 18 20:34:21.633064 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:34:21.633043    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9\": container with ID starting with 1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9 not found: ID does not exist" containerID="1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9"
Sep 18 20:34:21.633105 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.633069    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9"} err="failed to get container status \"1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9\": rpc error: code = NotFound desc = could not find container \"1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9\": container with ID starting with 1d9289497052a07126822580ee0b09fcfb8df89fc1523adacf0279d414f8f2a9 not found: ID does not exist"
Sep 18 20:34:21.635736 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:21.635716    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6w89x"]
Sep 18 20:34:22.722831 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:22.722798    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="848c20cf-6352-45fc-84ec-209370583788" path="/var/lib/kubelet/pods/848c20cf-6352-45fc-84ec-209370583788/volumes"
Sep 18 20:34:26.224860 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.224821    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:26.247443 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247413    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"]
Sep 18 20:34:26.247767 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247755    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kube-rbac-proxy"
Sep 18 20:34:26.247814 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247769    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kube-rbac-proxy"
Sep 18 20:34:26.247814 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247778    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="storage-initializer"
Sep 18 20:34:26.247814 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247784    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="storage-initializer"
Sep 18 20:34:26.247814 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247798    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kube-rbac-proxy"
Sep 18 20:34:26.247814 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247803    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kube-rbac-proxy"
Sep 18 20:34:26.247814 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247811    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container"
Sep 18 20:34:26.247814 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247816    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container"
Sep 18 20:34:26.248021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247826    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container"
Sep 18 20:34:26.248021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247831    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container"
Sep 18 20:34:26.248021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247840    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="storage-initializer"
Sep 18 20:34:26.248021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247845    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="storage-initializer"
Sep 18 20:34:26.248021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247899    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kserve-container"
Sep 18 20:34:26.248021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247908    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kserve-container"
Sep 18 20:34:26.248021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247915    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40eccc2-afa9-41d3-b828-d91a8eb096be" containerName="kube-rbac-proxy"
Sep 18 20:34:26.248021 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.247921    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="848c20cf-6352-45fc-84ec-209370583788" containerName="kube-rbac-proxy"
Sep 18 20:34:26.251215 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.251198    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.253439 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.253412    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-878aa9-predictor-serving-cert\""
Sep 18 20:34:26.253598 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.253496    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-878aa9-kube-rbac-proxy-sar-config\""
Sep 18 20:34:26.253598 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.253503    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:34:26.253598 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.253521    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:34:26.253812 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.253720    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ttgfx\""
Sep 18 20:34:26.253812 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.253753    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:34:26.253908 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.253812    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:34:26.261096 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.261075    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"]
Sep 18 20:34:26.305908 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.305884    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-cabundle-cert\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.306054 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.305914    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da9e010-162a-4787-a885-32df752e2d3f-proxy-tls\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.306054 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.305945    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thcdn\" (UniqueName: \"kubernetes.io/projected/5da9e010-162a-4787-a885-32df752e2d3f-kube-api-access-thcdn\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.306054 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.306031    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5da9e010-162a-4787-a885-32df752e2d3f-kserve-provision-location\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.306203 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.306084    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.406917 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.406882    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.407082 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.406952    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-cabundle-cert\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.407082 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.406979    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da9e010-162a-4787-a885-32df752e2d3f-proxy-tls\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.407082 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.407021    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thcdn\" (UniqueName: \"kubernetes.io/projected/5da9e010-162a-4787-a885-32df752e2d3f-kube-api-access-thcdn\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.407082 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.407057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5da9e010-162a-4787-a885-32df752e2d3f-kserve-provision-location\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.407504 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.407483    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5da9e010-162a-4787-a885-32df752e2d3f-kserve-provision-location\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.407875 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.407846    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.407875 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.407864    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-cabundle-cert\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.409535 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.409517    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da9e010-162a-4787-a885-32df752e2d3f-proxy-tls\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.414699 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.414676    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thcdn\" (UniqueName: \"kubernetes.io/projected/5da9e010-162a-4787-a885-32df752e2d3f-kube-api-access-thcdn\") pod \"isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") " pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.443496 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.443466    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:26.562652 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.562573    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:26.833400 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:26.833377    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"]
Sep 18 20:34:26.835138 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:34:26.835114    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da9e010_162a_4787_a885_32df752e2d3f.slice/crio-07d9103b837546258fe83b608ebbe5610270625924fff24a85710b4d79c43022 WatchSource:0}: Error finding container 07d9103b837546258fe83b608ebbe5610270625924fff24a85710b4d79c43022: Status 404 returned error can't find the container with id 07d9103b837546258fe83b608ebbe5610270625924fff24a85710b4d79c43022
Sep 18 20:34:27.645487 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:27.645431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" event={"ID":"5da9e010-162a-4787-a885-32df752e2d3f","Type":"ContainerStarted","Data":"09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7"}
Sep 18 20:34:27.645862 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:27.645521    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" event={"ID":"5da9e010-162a-4787-a885-32df752e2d3f","Type":"ContainerStarted","Data":"07d9103b837546258fe83b608ebbe5610270625924fff24a85710b4d79c43022"}
Sep 18 20:34:31.665307 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:31.665269    2557 generic.go:356] "Generic (PLEG): container finished" podID="5da9e010-162a-4787-a885-32df752e2d3f" containerID="09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7" exitCode=0
Sep 18 20:34:31.665711 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:31.665343    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" event={"ID":"5da9e010-162a-4787-a885-32df752e2d3f","Type":"ContainerDied","Data":"09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7"}
Sep 18 20:34:32.673171 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:32.673135    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" event={"ID":"5da9e010-162a-4787-a885-32df752e2d3f","Type":"ContainerStarted","Data":"9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18"}
Sep 18 20:34:32.673674 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:32.673211    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" event={"ID":"5da9e010-162a-4787-a885-32df752e2d3f","Type":"ContainerStarted","Data":"307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7"}
Sep 18 20:34:32.673674 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:32.673438    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:32.713014 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:32.712967    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podStartSLOduration=2.833508168 podStartE2EDuration="6.712953465s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.879445297s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:34:32.712048277 +0000 UTC m=+836.205401005" watchObservedRunningTime="2026-09-18 20:34:32.712953465 +0000 UTC m=+836.206306042"
Sep 18 20:34:33.678830 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:33.678798    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:33.680109 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:33.680079    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:34:34.683142 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:34.683106    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:34:39.687771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:39.687738    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:34:39.688301 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:39.688272    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:34:49.688309 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:49.688271    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:34:51.544630 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:51.544598    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:52.074768 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:52.074729    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:59.689089 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:34:59.689045    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:35:01.595588 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.595551    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:01.695041 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.695002    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:01.797986 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.797947    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:01.798823 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.798800    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"]
Sep 18 20:35:01.802372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.802349    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:01.804259 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.804234    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-ad25b-kube-rbac-proxy-sar-config\""
Sep 18 20:35:01.804403 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.804387    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-ad25b-predictor-serving-cert\""
Sep 18 20:35:01.813164 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.813144    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"]
Sep 18 20:35:01.862769 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.862706    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:01.903852 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.903817    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") " pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:01.903852 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.903854    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd699dc1-e09d-46eb-9e24-ba341010e1ce-proxy-tls\") pod \"message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") " pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:01.904126 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.903957    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd7c\" (UniqueName: \"kubernetes.io/projected/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-api-access-dwd7c\") pod \"message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") " pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:01.940799 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:01.940765    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:02.005507 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:02.005474    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") " pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:02.005654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:02.005520    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd699dc1-e09d-46eb-9e24-ba341010e1ce-proxy-tls\") pod \"message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") " pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:02.005654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:02.005583    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd7c\" (UniqueName: \"kubernetes.io/projected/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-api-access-dwd7c\") pod \"message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") " pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:02.006096 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:02.006070    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") " pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:02.007976 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:02.007954    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd699dc1-e09d-46eb-9e24-ba341010e1ce-proxy-tls\") pod \"message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") " pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:02.013375 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:02.013347    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd7c\" (UniqueName: \"kubernetes.io/projected/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-api-access-dwd7c\") pod \"message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") " pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:02.113857 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:02.113789    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:02.383957 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:02.383933    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"]
Sep 18 20:35:02.385989 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:35:02.385957    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd699dc1_e09d_46eb_9e24_ba341010e1ce.slice/crio-b96a6c28f1f50747d01ed07f1f3cde7a3f5ee1fd3c6001cdfec0175177809bc4 WatchSource:0}: Error finding container b96a6c28f1f50747d01ed07f1f3cde7a3f5ee1fd3c6001cdfec0175177809bc4: Status 404 returned error can't find the container with id b96a6c28f1f50747d01ed07f1f3cde7a3f5ee1fd3c6001cdfec0175177809bc4
Sep 18 20:35:02.387960 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:02.387943    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:35:02.809215 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:02.809180    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9" event={"ID":"dd699dc1-e09d-46eb-9e24-ba341010e1ce","Type":"ContainerStarted","Data":"b96a6c28f1f50747d01ed07f1f3cde7a3f5ee1fd3c6001cdfec0175177809bc4"}
Sep 18 20:35:03.815739 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:03.815659    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9" event={"ID":"dd699dc1-e09d-46eb-9e24-ba341010e1ce","Type":"ContainerStarted","Data":"9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc"}
Sep 18 20:35:03.816103 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:03.815747    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9" event={"ID":"dd699dc1-e09d-46eb-9e24-ba341010e1ce","Type":"ContainerStarted","Data":"c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90"}
Sep 18 20:35:03.816103 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:03.815774    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:03.816103 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:03.815802    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:03.817653 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:03.817631    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:03.833426 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:03.833380    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9" podStartSLOduration=1.715209369 podStartE2EDuration="2.833366811s" totalImagesPullingTime="1.118157442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:35:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:35:03.831643223 +0000 UTC m=+867.324995805" watchObservedRunningTime="2026-09-18 20:35:03.833366811 +0000 UTC m=+867.326719388"
Sep 18 20:35:09.689130 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:09.689088    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:35:10.829431 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:10.829402    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:35:11.774511 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:11.774478    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:11.900274 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:11.900236    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:19.688780 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:19.688688    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:35:29.688508 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:29.688442    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:35:38.670037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:38.670010    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:35:38.671887 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:38.671866    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:35:39.688896 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:39.688858    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:35:42.059760 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:42.059722    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:49.689634 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:35:49.689602    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:36:11.405860 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:11.405818    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:12.947739 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:12.947704    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:15.954942 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:15.954909    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:16.016696 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:16.016660    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:29.479085 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:29.479054    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"]
Sep 18 20:36:29.479928 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:29.479892    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" containerID="cri-o://307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7" gracePeriod=30
Sep 18 20:36:29.480102 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:29.479911    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kube-rbac-proxy" containerID="cri-o://9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18" gracePeriod=30
Sep 18 20:36:29.480102 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:29.479956    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:29.540475 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:29.540425    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:29.683484 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:29.683433    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 20:36:29.688749 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:29.688728    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:36:30.212557 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:30.212522    2557 generic.go:356] "Generic (PLEG): container finished" podID="5da9e010-162a-4787-a885-32df752e2d3f" containerID="9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18" exitCode=2
Sep 18 20:36:30.212720 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:30.212591    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" event={"ID":"5da9e010-162a-4787-a885-32df752e2d3f","Type":"ContainerDied","Data":"9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18"}
Sep 18 20:36:33.947976 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:33.947948    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:36:34.070261 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.070193    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thcdn\" (UniqueName: \"kubernetes.io/projected/5da9e010-162a-4787-a885-32df752e2d3f-kube-api-access-thcdn\") pod \"5da9e010-162a-4787-a885-32df752e2d3f\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") "
Sep 18 20:36:34.070261 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.070236    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-cabundle-cert\") pod \"5da9e010-162a-4787-a885-32df752e2d3f\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") "
Sep 18 20:36:34.070261 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.070257    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-kube-rbac-proxy-sar-config\") pod \"5da9e010-162a-4787-a885-32df752e2d3f\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") "
Sep 18 20:36:34.070554 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.070284    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5da9e010-162a-4787-a885-32df752e2d3f-kserve-provision-location\") pod \"5da9e010-162a-4787-a885-32df752e2d3f\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") "
Sep 18 20:36:34.070554 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.070311    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da9e010-162a-4787-a885-32df752e2d3f-proxy-tls\") pod \"5da9e010-162a-4787-a885-32df752e2d3f\" (UID: \"5da9e010-162a-4787-a885-32df752e2d3f\") "
Sep 18 20:36:34.070667 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.070626    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5da9e010-162a-4787-a885-32df752e2d3f" (UID: "5da9e010-162a-4787-a885-32df752e2d3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:36:34.070667 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.070635    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5da9e010-162a-4787-a885-32df752e2d3f" (UID: "5da9e010-162a-4787-a885-32df752e2d3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:36:34.070744 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.070640    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da9e010-162a-4787-a885-32df752e2d3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5da9e010-162a-4787-a885-32df752e2d3f" (UID: "5da9e010-162a-4787-a885-32df752e2d3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:36:34.072182 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.072160    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da9e010-162a-4787-a885-32df752e2d3f-kube-api-access-thcdn" (OuterVolumeSpecName: "kube-api-access-thcdn") pod "5da9e010-162a-4787-a885-32df752e2d3f" (UID: "5da9e010-162a-4787-a885-32df752e2d3f"). InnerVolumeSpecName "kube-api-access-thcdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:36:34.072331 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.072309    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da9e010-162a-4787-a885-32df752e2d3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5da9e010-162a-4787-a885-32df752e2d3f" (UID: "5da9e010-162a-4787-a885-32df752e2d3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:36:34.171788 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.171766    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thcdn\" (UniqueName: \"kubernetes.io/projected/5da9e010-162a-4787-a885-32df752e2d3f-kube-api-access-thcdn\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:36:34.171788 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.171787    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-cabundle-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:36:34.171960 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.171798    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5da9e010-162a-4787-a885-32df752e2d3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:36:34.171960 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.171806    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5da9e010-162a-4787-a885-32df752e2d3f-kserve-provision-location\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:36:34.171960 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.171815    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da9e010-162a-4787-a885-32df752e2d3f-proxy-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:36:34.233547 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.233511    2557 generic.go:356] "Generic (PLEG): container finished" podID="5da9e010-162a-4787-a885-32df752e2d3f" containerID="307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7" exitCode=0
Sep 18 20:36:34.233696 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.233571    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"
Sep 18 20:36:34.233696 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.233591    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" event={"ID":"5da9e010-162a-4787-a885-32df752e2d3f","Type":"ContainerDied","Data":"307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7"}
Sep 18 20:36:34.233696 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.233651    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb" event={"ID":"5da9e010-162a-4787-a885-32df752e2d3f","Type":"ContainerDied","Data":"07d9103b837546258fe83b608ebbe5610270625924fff24a85710b4d79c43022"}
Sep 18 20:36:34.233696 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.233679    2557 scope.go:117] "RemoveContainer" containerID="9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18"
Sep 18 20:36:34.242958 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.242939    2557 scope.go:117] "RemoveContainer" containerID="307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7"
Sep 18 20:36:34.250151 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.250133    2557 scope.go:117] "RemoveContainer" containerID="09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7"
Sep 18 20:36:34.256055 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.256033    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"]
Sep 18 20:36:34.257657 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.257640    2557 scope.go:117] "RemoveContainer" containerID="9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18"
Sep 18 20:36:34.257960 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:36:34.257934    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18\": container with ID starting with 9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18 not found: ID does not exist" containerID="9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18"
Sep 18 20:36:34.258068 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.257972    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18"} err="failed to get container status \"9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18\": rpc error: code = NotFound desc = could not find container \"9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18\": container with ID starting with 9b6e63ba8f1f04c8365fc6b76c0e6dc611dbc3c4dc19259fadd2a02210cacc18 not found: ID does not exist"
Sep 18 20:36:34.258068 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.257996    2557 scope.go:117] "RemoveContainer" containerID="307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7"
Sep 18 20:36:34.258874 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:36:34.258326    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7\": container with ID starting with 307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7 not found: ID does not exist" containerID="307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7"
Sep 18 20:36:34.258874 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.258359    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7"} err="failed to get container status \"307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7\": rpc error: code = NotFound desc = could not find container \"307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7\": container with ID starting with 307f020bf4661ba74b0091cc6fe008511e8d301dcb316265385aba1027f70df7 not found: ID does not exist"
Sep 18 20:36:34.258874 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.258381    2557 scope.go:117] "RemoveContainer" containerID="09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7"
Sep 18 20:36:34.258874 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:36:34.258678    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7\": container with ID starting with 09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7 not found: ID does not exist" containerID="09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7"
Sep 18 20:36:34.258874 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.258703    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7"} err="failed to get container status \"09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7\": rpc error: code = NotFound desc = could not find container \"09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7\": container with ID starting with 09bf644a783f5635e835f2ff581286c132340d637ac37e7945047a2a276407b7 not found: ID does not exist"
Sep 18 20:36:34.260247 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.260230    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-878aa9-predictor-6d5d6b4759-wpxcb"]
Sep 18 20:36:34.675995 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.675962    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:34.722629 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.722602    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da9e010-162a-4787-a885-32df752e2d3f" path="/var/lib/kubelet/pods/5da9e010-162a-4787-a885-32df752e2d3f/volumes"
Sep 18 20:36:34.937876 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:34.937849    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:44.702100 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:44.702062    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:47.257009 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.256976    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9_dd699dc1-e09d-46eb-9e24-ba341010e1ce/kserve-container/0.log"
Sep 18 20:36:47.480003 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.479968    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"]
Sep 18 20:36:47.480227 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.480204    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9" podUID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" containerName="kserve-container" containerID="cri-o://c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90" gracePeriod=30
Sep 18 20:36:47.480299 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.480241    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9" podUID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" containerName="kube-rbac-proxy" containerID="cri-o://9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc" gracePeriod=30
Sep 18 20:36:47.482675 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.482648    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:47.579587 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.579501    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:47.691989 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.691954    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:47.740394 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.740373    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:36:47.884162 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.884084    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwd7c\" (UniqueName: \"kubernetes.io/projected/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-api-access-dwd7c\") pod \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") "
Sep 18 20:36:47.884162 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.884138    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-rbac-proxy-sar-config\") pod \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") "
Sep 18 20:36:47.884378 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.884190    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd699dc1-e09d-46eb-9e24-ba341010e1ce-proxy-tls\") pod \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\" (UID: \"dd699dc1-e09d-46eb-9e24-ba341010e1ce\") "
Sep 18 20:36:47.884525 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.884504    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd699dc1-e09d-46eb-9e24-ba341010e1ce" (UID: "dd699dc1-e09d-46eb-9e24-ba341010e1ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:36:47.886213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.886189    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd699dc1-e09d-46eb-9e24-ba341010e1ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd699dc1-e09d-46eb-9e24-ba341010e1ce" (UID: "dd699dc1-e09d-46eb-9e24-ba341010e1ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:36:47.886213 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.886199    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-api-access-dwd7c" (OuterVolumeSpecName: "kube-api-access-dwd7c") pod "dd699dc1-e09d-46eb-9e24-ba341010e1ce" (UID: "dd699dc1-e09d-46eb-9e24-ba341010e1ce"). InnerVolumeSpecName "kube-api-access-dwd7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:36:47.985243 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.985208    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:36:47.985243 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.985240    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd699dc1-e09d-46eb-9e24-ba341010e1ce-proxy-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:36:47.985243 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:47.985250    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwd7c\" (UniqueName: \"kubernetes.io/projected/dd699dc1-e09d-46eb-9e24-ba341010e1ce-kube-api-access-dwd7c\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:36:48.296297 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.296260    2557 generic.go:356] "Generic (PLEG): container finished" podID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" containerID="9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc" exitCode=2
Sep 18 20:36:48.296297 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.296292    2557 generic.go:356] "Generic (PLEG): container finished" podID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" containerID="c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90" exitCode=2
Sep 18 20:36:48.296778 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.296316    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"
Sep 18 20:36:48.296778 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.296320    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9" event={"ID":"dd699dc1-e09d-46eb-9e24-ba341010e1ce","Type":"ContainerDied","Data":"9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc"}
Sep 18 20:36:48.296778 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.296380    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9" event={"ID":"dd699dc1-e09d-46eb-9e24-ba341010e1ce","Type":"ContainerDied","Data":"c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90"}
Sep 18 20:36:48.296778 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.296413    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9" event={"ID":"dd699dc1-e09d-46eb-9e24-ba341010e1ce","Type":"ContainerDied","Data":"b96a6c28f1f50747d01ed07f1f3cde7a3f5ee1fd3c6001cdfec0175177809bc4"}
Sep 18 20:36:48.296778 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.296467    2557 scope.go:117] "RemoveContainer" containerID="9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc"
Sep 18 20:36:48.304881 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.304865    2557 scope.go:117] "RemoveContainer" containerID="c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90"
Sep 18 20:36:48.312143 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.312126    2557 scope.go:117] "RemoveContainer" containerID="9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc"
Sep 18 20:36:48.312401 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:36:48.312385    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc\": container with ID starting with 9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc not found: ID does not exist" containerID="9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc"
Sep 18 20:36:48.312449 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.312410    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc"} err="failed to get container status \"9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc\": rpc error: code = NotFound desc = could not find container \"9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc\": container with ID starting with 9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc not found: ID does not exist"
Sep 18 20:36:48.312449 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.312425    2557 scope.go:117] "RemoveContainer" containerID="c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90"
Sep 18 20:36:48.312665 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:36:48.312645    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90\": container with ID starting with c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90 not found: ID does not exist" containerID="c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90"
Sep 18 20:36:48.312701 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.312669    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90"} err="failed to get container status \"c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90\": rpc error: code = NotFound desc = could not find container \"c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90\": container with ID starting with c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90 not found: ID does not exist"
Sep 18 20:36:48.312701 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.312682    2557 scope.go:117] "RemoveContainer" containerID="9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc"
Sep 18 20:36:48.312900 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.312881    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc"} err="failed to get container status \"9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc\": rpc error: code = NotFound desc = could not find container \"9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc\": container with ID starting with 9db96d3cda3d8ed023100f79444c430eef41b1e042826d7a62a2a29fca639ddc not found: ID does not exist"
Sep 18 20:36:48.312944 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.312902    2557 scope.go:117] "RemoveContainer" containerID="c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90"
Sep 18 20:36:48.313132 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.313112    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90"} err="failed to get container status \"c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90\": rpc error: code = NotFound desc = could not find container \"c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90\": container with ID starting with c5bd916bb6ec25d5875a5d660de19a457cf15aca86b3ee84529139025815bb90 not found: ID does not exist"
Sep 18 20:36:48.317499 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.317479    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"]
Sep 18 20:36:48.321539 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.321521    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ad25b-predictor-6f575f5b9d-54jv9"]
Sep 18 20:36:48.722944 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:48.722911    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" path="/var/lib/kubelet/pods/dd699dc1-e09d-46eb-9e24-ba341010e1ce/volumes"
Sep 18 20:36:49.896429 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.896399    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:49.928632 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.928601    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"]
Sep 18 20:36:49.928943 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.928931    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" containerName="kserve-container"
Sep 18 20:36:49.928991 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.928944    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" containerName="kserve-container"
Sep 18 20:36:49.928991 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.928959    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kube-rbac-proxy"
Sep 18 20:36:49.928991 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.928965    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kube-rbac-proxy"
Sep 18 20:36:49.928991 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.928972    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="storage-initializer"
Sep 18 20:36:49.928991 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.928979    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="storage-initializer"
Sep 18 20:36:49.928991 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.928985    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container"
Sep 18 20:36:49.928991 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.928991    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container"
Sep 18 20:36:49.929226 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.929002    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" containerName="kube-rbac-proxy"
Sep 18 20:36:49.929226 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.929007    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" containerName="kube-rbac-proxy"
Sep 18 20:36:49.929226 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.929062    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kube-rbac-proxy"
Sep 18 20:36:49.929226 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.929069    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da9e010-162a-4787-a885-32df752e2d3f" containerName="kserve-container"
Sep 18 20:36:49.929226 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.929079    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" containerName="kserve-container"
Sep 18 20:36:49.929226 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.929084    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd699dc1-e09d-46eb-9e24-ba341010e1ce" containerName="kube-rbac-proxy"
Sep 18 20:36:49.933825 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.933804    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:49.936037 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.936016    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:36:49.936157 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.936036    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-5d7b5-predictor-serving-cert\""
Sep 18 20:36:49.936157 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.936079    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:36:49.936157 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.936079    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ttgfx\""
Sep 18 20:36:49.936293 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.936216    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:36:49.936356 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.936342    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:36:49.936420 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.936407    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-5d7b5-kube-rbac-proxy-sar-config\""
Sep 18 20:36:49.943838 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:49.943812    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"]
Sep 18 20:36:50.016351 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.016306    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:50.101789 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.101756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.101951 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.101814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a871a8f-5085-4d62-9222-2ae97bb8587a-kserve-provision-location\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.101951 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.101852    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a871a8f-5085-4d62-9222-2ae97bb8587a-proxy-tls\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.101951 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.101872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c2kc\" (UniqueName: \"kubernetes.io/projected/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-api-access-2c2kc\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.102062 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.101962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-cabundle-cert\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.202898 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.202869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-cabundle-cert\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.203025 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.202902    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.203025 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.202942    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a871a8f-5085-4d62-9222-2ae97bb8587a-kserve-provision-location\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.203025 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.202964    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a871a8f-5085-4d62-9222-2ae97bb8587a-proxy-tls\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.203025 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.202980    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c2kc\" (UniqueName: \"kubernetes.io/projected/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-api-access-2c2kc\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.203404 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.203378    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a871a8f-5085-4d62-9222-2ae97bb8587a-kserve-provision-location\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.203558 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.203541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.203635 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.203619    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-cabundle-cert\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.205502 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.205480    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a871a8f-5085-4d62-9222-2ae97bb8587a-proxy-tls\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.212101 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.212083    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c2kc\" (UniqueName: \"kubernetes.io/projected/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-api-access-2c2kc\") pod \"raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") " pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.245162 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.245140    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:50.516052 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:50.516026    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"]
Sep 18 20:36:50.518482 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:36:50.518437    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a871a8f_5085_4d62_9222_2ae97bb8587a.slice/crio-7fe5a0be232f8646e61d56c04c5ab1aa1930f40950e7a2b06e9754d881212cdd WatchSource:0}: Error finding container 7fe5a0be232f8646e61d56c04c5ab1aa1930f40950e7a2b06e9754d881212cdd: Status 404 returned error can't find the container with id 7fe5a0be232f8646e61d56c04c5ab1aa1930f40950e7a2b06e9754d881212cdd
Sep 18 20:36:51.313261 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:51.313230    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" event={"ID":"3a871a8f-5085-4d62-9222-2ae97bb8587a","Type":"ContainerStarted","Data":"61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf"}
Sep 18 20:36:51.313631 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:51.313283    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" event={"ID":"3a871a8f-5085-4d62-9222-2ae97bb8587a","Type":"ContainerStarted","Data":"7fe5a0be232f8646e61d56c04c5ab1aa1930f40950e7a2b06e9754d881212cdd"}
Sep 18 20:36:55.332721 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:55.332684    2557 generic.go:356] "Generic (PLEG): container finished" podID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerID="61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf" exitCode=0
Sep 18 20:36:55.333070 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:55.332758    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" event={"ID":"3a871a8f-5085-4d62-9222-2ae97bb8587a","Type":"ContainerDied","Data":"61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf"}
Sep 18 20:36:56.339394 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:56.339358    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" event={"ID":"3a871a8f-5085-4d62-9222-2ae97bb8587a","Type":"ContainerStarted","Data":"71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011"}
Sep 18 20:36:56.339780 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:56.339445    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" event={"ID":"3a871a8f-5085-4d62-9222-2ae97bb8587a","Type":"ContainerStarted","Data":"7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473"}
Sep 18 20:36:56.339780 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:56.339644    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:56.359572 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:56.359504    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podStartSLOduration=3.401474406 podStartE2EDuration="7.359441225s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.957966819s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:36:56.359064518 +0000 UTC m=+979.852417099" watchObservedRunningTime="2026-09-18 20:36:56.359441225 +0000 UTC m=+979.852793804"
Sep 18 20:36:57.344047 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:57.344013    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:36:57.345137 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:57.345104    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:36:58.348505 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:36:58.348441    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:37:03.353411 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:37:03.353382    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:37:03.354012 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:37:03.353983    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:37:13.354466 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:37:13.354422    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:37:23.354501 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:37:23.354444    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:37:32.779505 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:37:32.779470    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:37:33.354089 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:37:33.354049    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:37:43.354267 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:37:43.354230    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:37:53.354273 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:37:53.354225    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:38:03.354349 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:03.354299    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:38:13.355584 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:13.355546    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:38:20.505285 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:20.505208    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"]
Sep 18 20:38:20.505780 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:20.505479    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" containerID="cri-o://7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473" gracePeriod=30
Sep 18 20:38:20.505780 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:20.505533    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kube-rbac-proxy" containerID="cri-o://71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011" gracePeriod=30
Sep 18 20:38:20.529734 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:20.529701    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:20.729162 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:20.729133    2557 generic.go:356] "Generic (PLEG): container finished" podID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerID="71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011" exitCode=2
Sep 18 20:38:20.729316 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:20.729204    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" event={"ID":"3a871a8f-5085-4d62-9222-2ae97bb8587a","Type":"ContainerDied","Data":"71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011"}
Sep 18 20:38:23.349531 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:23.349493    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 18 20:38:23.354034 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:23.354003    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:38:25.052440 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.052416    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:38:25.156244 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.156174    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a871a8f-5085-4d62-9222-2ae97bb8587a-kserve-provision-location\") pod \"3a871a8f-5085-4d62-9222-2ae97bb8587a\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") "
Sep 18 20:38:25.156244 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.156207    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c2kc\" (UniqueName: \"kubernetes.io/projected/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-api-access-2c2kc\") pod \"3a871a8f-5085-4d62-9222-2ae97bb8587a\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") "
Sep 18 20:38:25.156484 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.156243    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a871a8f-5085-4d62-9222-2ae97bb8587a-proxy-tls\") pod \"3a871a8f-5085-4d62-9222-2ae97bb8587a\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") "
Sep 18 20:38:25.156484 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.156285    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-rbac-proxy-sar-config\") pod \"3a871a8f-5085-4d62-9222-2ae97bb8587a\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") "
Sep 18 20:38:25.156484 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.156330    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-cabundle-cert\") pod \"3a871a8f-5085-4d62-9222-2ae97bb8587a\" (UID: \"3a871a8f-5085-4d62-9222-2ae97bb8587a\") "
Sep 18 20:38:25.156654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.156518    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a871a8f-5085-4d62-9222-2ae97bb8587a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a871a8f-5085-4d62-9222-2ae97bb8587a" (UID: "3a871a8f-5085-4d62-9222-2ae97bb8587a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:38:25.156715 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.156689    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a871a8f-5085-4d62-9222-2ae97bb8587a-kserve-provision-location\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:38:25.156766 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.156723    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a871a8f-5085-4d62-9222-2ae97bb8587a" (UID: "3a871a8f-5085-4d62-9222-2ae97bb8587a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:38:25.156766 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.156754    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a871a8f-5085-4d62-9222-2ae97bb8587a" (UID: "3a871a8f-5085-4d62-9222-2ae97bb8587a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:38:25.158316 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.158294    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a871a8f-5085-4d62-9222-2ae97bb8587a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a871a8f-5085-4d62-9222-2ae97bb8587a" (UID: "3a871a8f-5085-4d62-9222-2ae97bb8587a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:38:25.158371 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.158338    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-api-access-2c2kc" (OuterVolumeSpecName: "kube-api-access-2c2kc") pod "3a871a8f-5085-4d62-9222-2ae97bb8587a" (UID: "3a871a8f-5085-4d62-9222-2ae97bb8587a"). InnerVolumeSpecName "kube-api-access-2c2kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:38:25.257692 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.257665    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c2kc\" (UniqueName: \"kubernetes.io/projected/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-api-access-2c2kc\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:38:25.257817 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.257705    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a871a8f-5085-4d62-9222-2ae97bb8587a-proxy-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:38:25.257817 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.257722    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:38:25.257817 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.257732    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a871a8f-5085-4d62-9222-2ae97bb8587a-cabundle-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:38:25.655499 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.655442    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:25.709631 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.709600    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"]
Sep 18 20:38:25.709971 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.709957    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kube-rbac-proxy"
Sep 18 20:38:25.710020 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.709975    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kube-rbac-proxy"
Sep 18 20:38:25.710020 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.709988    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container"
Sep 18 20:38:25.710020 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.709994    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container"
Sep 18 20:38:25.710020 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.710006    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="storage-initializer"
Sep 18 20:38:25.710020 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.710011    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="storage-initializer"
Sep 18 20:38:25.710179 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.710075    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kube-rbac-proxy"
Sep 18 20:38:25.710179 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.710085    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerName="kserve-container"
Sep 18 20:38:25.713302 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.713285    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.715501 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.715480    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-89cb6-predictor-serving-cert\""
Sep 18 20:38:25.715587 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.715573    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-89cb6-kube-rbac-proxy-sar-config\""
Sep 18 20:38:25.725183 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.725160    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"]
Sep 18 20:38:25.754429 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.754403    2557 generic.go:356] "Generic (PLEG): container finished" podID="3a871a8f-5085-4d62-9222-2ae97bb8587a" containerID="7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473" exitCode=0
Sep 18 20:38:25.754582 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.754481    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"
Sep 18 20:38:25.754582 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.754506    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" event={"ID":"3a871a8f-5085-4d62-9222-2ae97bb8587a","Type":"ContainerDied","Data":"7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473"}
Sep 18 20:38:25.754582 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.754570    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92" event={"ID":"3a871a8f-5085-4d62-9222-2ae97bb8587a","Type":"ContainerDied","Data":"7fe5a0be232f8646e61d56c04c5ab1aa1930f40950e7a2b06e9754d881212cdd"}
Sep 18 20:38:25.754692 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.754611    2557 scope.go:117] "RemoveContainer" containerID="71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011"
Sep 18 20:38:25.761968 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.761943    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-cabundle-cert\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.762075 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.761989    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade359b5-6135-45a0-b967-0690b6b9fbbc-kserve-provision-location\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.762129 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.762079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.762182 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.762151    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade359b5-6135-45a0-b967-0690b6b9fbbc-proxy-tls\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.762231 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.762181    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q67m\" (UniqueName: \"kubernetes.io/projected/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-api-access-4q67m\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.764390 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.764374    2557 scope.go:117] "RemoveContainer" containerID="7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473"
Sep 18 20:38:25.772038 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.772020    2557 scope.go:117] "RemoveContainer" containerID="61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf"
Sep 18 20:38:25.777098 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.777065    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"]
Sep 18 20:38:25.779097 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.779078    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5d7b5-predictor-78fb84d45d-mtl92"]
Sep 18 20:38:25.780254 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.780236    2557 scope.go:117] "RemoveContainer" containerID="71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011"
Sep 18 20:38:25.780539 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:38:25.780519    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011\": container with ID starting with 71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011 not found: ID does not exist" containerID="71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011"
Sep 18 20:38:25.780648 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.780545    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011"} err="failed to get container status \"71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011\": rpc error: code = NotFound desc = could not find container \"71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011\": container with ID starting with 71aaf45f5cc98585d90818860a4ecdef38e8d2d78962bac3b122676006f23011 not found: ID does not exist"
Sep 18 20:38:25.780648 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.780562    2557 scope.go:117] "RemoveContainer" containerID="7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473"
Sep 18 20:38:25.780809 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:38:25.780796    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473\": container with ID starting with 7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473 not found: ID does not exist" containerID="7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473"
Sep 18 20:38:25.780851 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.780812    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473"} err="failed to get container status \"7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473\": rpc error: code = NotFound desc = could not find container \"7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473\": container with ID starting with 7e49da09ec461c4826878852371cae522dac3a429f5879d9782c18008ce14473 not found: ID does not exist"
Sep 18 20:38:25.780851 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.780823    2557 scope.go:117] "RemoveContainer" containerID="61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf"
Sep 18 20:38:25.781061 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:38:25.781044    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf\": container with ID starting with 61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf not found: ID does not exist" containerID="61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf"
Sep 18 20:38:25.781111 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.781066    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf"} err="failed to get container status \"61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf\": rpc error: code = NotFound desc = could not find container \"61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf\": container with ID starting with 61b402e442ffb104b824e8e1d7953a4f48337e0bce5c07128889e185d6f71caf not found: ID does not exist"
Sep 18 20:38:25.863121 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.863091    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade359b5-6135-45a0-b967-0690b6b9fbbc-proxy-tls\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.863287 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.863130    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q67m\" (UniqueName: \"kubernetes.io/projected/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-api-access-4q67m\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.863287 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.863197    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-cabundle-cert\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.863287 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.863237    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade359b5-6135-45a0-b967-0690b6b9fbbc-kserve-provision-location\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.863287 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:38:25.863275    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-serving-cert: secret "raw-sklearn-runtime-89cb6-predictor-serving-cert" not found
Sep 18 20:38:25.863562 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.863297    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.863562 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:38:25.863355    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade359b5-6135-45a0-b967-0690b6b9fbbc-proxy-tls podName:ade359b5-6135-45a0-b967-0690b6b9fbbc nodeName:}" failed. No retries permitted until 2026-09-18 20:38:26.363333232 +0000 UTC m=+1069.856685795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ade359b5-6135-45a0-b967-0690b6b9fbbc-proxy-tls") pod "raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" (UID: "ade359b5-6135-45a0-b967-0690b6b9fbbc") : secret "raw-sklearn-runtime-89cb6-predictor-serving-cert" not found
Sep 18 20:38:25.863725 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.863702    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade359b5-6135-45a0-b967-0690b6b9fbbc-kserve-provision-location\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.863967 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.863948    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-cabundle-cert\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.864075 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.864059    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:25.870988 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:25.870971    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q67m\" (UniqueName: \"kubernetes.io/projected/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-api-access-4q67m\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:26.015347 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:26.015308    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:26.368149 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:26.368075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade359b5-6135-45a0-b967-0690b6b9fbbc-proxy-tls\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:26.370537 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:26.370513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade359b5-6135-45a0-b967-0690b6b9fbbc-proxy-tls\") pod \"raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:26.624402 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:26.624318    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:26.723611 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:26.723580    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a871a8f-5085-4d62-9222-2ae97bb8587a" path="/var/lib/kubelet/pods/3a871a8f-5085-4d62-9222-2ae97bb8587a/volumes"
Sep 18 20:38:26.922995 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:26.922958    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"]
Sep 18 20:38:26.925359 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:38:26.925332    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade359b5_6135_45a0_b967_0690b6b9fbbc.slice/crio-e45ef6d22405f1383e7cc3fb24f16053966cd4e25e774800f08a0829fd826ee4 WatchSource:0}: Error finding container e45ef6d22405f1383e7cc3fb24f16053966cd4e25e774800f08a0829fd826ee4: Status 404 returned error can't find the container with id e45ef6d22405f1383e7cc3fb24f16053966cd4e25e774800f08a0829fd826ee4
Sep 18 20:38:27.766443 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:27.766410    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" event={"ID":"ade359b5-6135-45a0-b967-0690b6b9fbbc","Type":"ContainerStarted","Data":"1d7bd584e5ee3050d248675efa6bc3c9f3803b9d26180db421eb4d56dd947383"}
Sep 18 20:38:27.766873 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:27.766476    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" event={"ID":"ade359b5-6135-45a0-b967-0690b6b9fbbc","Type":"ContainerStarted","Data":"e45ef6d22405f1383e7cc3fb24f16053966cd4e25e774800f08a0829fd826ee4"}
Sep 18 20:38:30.780204 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:30.780123    2557 generic.go:356] "Generic (PLEG): container finished" podID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerID="1d7bd584e5ee3050d248675efa6bc3c9f3803b9d26180db421eb4d56dd947383" exitCode=0
Sep 18 20:38:30.780582 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:30.780201    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" event={"ID":"ade359b5-6135-45a0-b967-0690b6b9fbbc","Type":"ContainerDied","Data":"1d7bd584e5ee3050d248675efa6bc3c9f3803b9d26180db421eb4d56dd947383"}
Sep 18 20:38:31.789847 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:31.789803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" event={"ID":"ade359b5-6135-45a0-b967-0690b6b9fbbc","Type":"ContainerStarted","Data":"55ae256761d304663e58d07832eaa4b70ceb6e0c1b6aa4f3caa48c27090c004c"}
Sep 18 20:38:31.790251 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:31.789874    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" event={"ID":"ade359b5-6135-45a0-b967-0690b6b9fbbc","Type":"ContainerStarted","Data":"525890e6f23eed865f7ea6ad9cde66c3fc22211122e1a3a1c5636f249c99399e"}
Sep 18 20:38:31.791991 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:31.791651    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:31.791991 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:31.791752    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:31.793148 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:31.793086    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:38:31.811238 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:31.811196    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podStartSLOduration=3.267455459 podStartE2EDuration="6.811181111s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.543725652s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:38:31.809158674 +0000 UTC m=+1075.302511247" watchObservedRunningTime="2026-09-18 20:38:31.811181111 +0000 UTC m=+1075.304533689"
Sep 18 20:38:32.795351 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:32.795310    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:38:37.799861 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:37.799832    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:38:37.800553 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:37.800523    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:38:47.801408 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:47.801370    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:38:57.735262 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:57.735226    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:57.800698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:57.800662    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:38:57.835974 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:38:57.835943    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:39:07.800407 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:07.800364    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:39:17.800822 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:17.800787    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:39:27.800408 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:27.800367    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:39:37.801078 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:37.801037    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:39:38.756381 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:38.756340    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:39:43.895077 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:43.895029    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:39:43.978320 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:43.978283    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"]
Sep 18 20:39:43.978576 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:43.978551    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:39:43.982179 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:43.982159    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:43.984192 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:43.984174    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 20:39:43.984281 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:43.984193    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 20:39:43.992667 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:43.992642    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"]
Sep 18 20:39:44.143517 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.143484    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df53336-693f-47d2-ae93-7eabcb1a416c-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.143712 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.143554    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.143712 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.143597    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.143845 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.143740    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btwv9\" (UniqueName: \"kubernetes.io/projected/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-api-access-btwv9\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.143845 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.143777    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df53336-693f-47d2-ae93-7eabcb1a416c-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.215181 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.215142    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:39:44.244984 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.244946    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df53336-693f-47d2-ae93-7eabcb1a416c-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.245167 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.244993    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.245167 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.245027    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.245167 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:39:44.245117    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 18 20:39:44.245167 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.245133    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btwv9\" (UniqueName: \"kubernetes.io/projected/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-api-access-btwv9\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.245167 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.245157    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df53336-693f-47d2-ae93-7eabcb1a416c-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.245445 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:39:44.245192    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df53336-693f-47d2-ae93-7eabcb1a416c-proxy-tls podName:4df53336-693f-47d2-ae93-7eabcb1a416c nodeName:}" failed. No retries permitted until 2026-09-18 20:39:44.745176838 +0000 UTC m=+1148.238529394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4df53336-693f-47d2-ae93-7eabcb1a416c-proxy-tls") pod "isvc-canary-create-predictor-76b74f4d6c-2f5vx" (UID: "4df53336-693f-47d2-ae93-7eabcb1a416c") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 18 20:39:44.245579 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.245558    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df53336-693f-47d2-ae93-7eabcb1a416c-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.245778 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.245758    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.245883 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.245864    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.255938 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.255912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btwv9\" (UniqueName: \"kubernetes.io/projected/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-api-access-btwv9\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.721378 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.721331    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:39:44.749506 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.749479    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df53336-693f-47d2-ae93-7eabcb1a416c-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.752070 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.752034    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df53336-693f-47d2-ae93-7eabcb1a416c-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-2f5vx\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:44.893388 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:44.893366    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:45.162917 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:45.162789    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"]
Sep 18 20:39:45.167280 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:39:45.167231    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df53336_693f_47d2_ae93_7eabcb1a416c.slice/crio-466a2e4a322b08454511f565eb4b5a8246262b203a470f9c1f36731f5c90b514 WatchSource:0}: Error finding container 466a2e4a322b08454511f565eb4b5a8246262b203a470f9c1f36731f5c90b514: Status 404 returned error can't find the container with id 466a2e4a322b08454511f565eb4b5a8246262b203a470f9c1f36731f5c90b514
Sep 18 20:39:46.136491 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:46.136434    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" event={"ID":"4df53336-693f-47d2-ae93-7eabcb1a416c","Type":"ContainerStarted","Data":"ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8"}
Sep 18 20:39:46.136654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:46.136522    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" event={"ID":"4df53336-693f-47d2-ae93-7eabcb1a416c","Type":"ContainerStarted","Data":"466a2e4a322b08454511f565eb4b5a8246262b203a470f9c1f36731f5c90b514"}
Sep 18 20:39:47.801357 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:47.801329    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:39:49.152917 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:49.152841    2557 generic.go:356] "Generic (PLEG): container finished" podID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerID="ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8" exitCode=0
Sep 18 20:39:49.152917 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:49.152889    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" event={"ID":"4df53336-693f-47d2-ae93-7eabcb1a416c","Type":"ContainerDied","Data":"ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8"}
Sep 18 20:39:50.160281 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:50.160241    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" event={"ID":"4df53336-693f-47d2-ae93-7eabcb1a416c","Type":"ContainerStarted","Data":"c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884"}
Sep 18 20:39:50.160686 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:50.160310    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" event={"ID":"4df53336-693f-47d2-ae93-7eabcb1a416c","Type":"ContainerStarted","Data":"2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498"}
Sep 18 20:39:50.160686 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:50.160485    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:50.180703 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:50.180659    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podStartSLOduration=3.426565197 podStartE2EDuration="7.180645192s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.754079995s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:39:50.178673633 +0000 UTC m=+1153.672026212" watchObservedRunningTime="2026-09-18 20:39:50.180645192 +0000 UTC m=+1153.673997771"
Sep 18 20:39:51.165694 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:51.165663    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:51.167089 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:51.167059    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:39:52.170446 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:52.170413    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:39:56.174561 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:56.174527    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"]
Sep 18 20:39:56.175023 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:56.174831    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" containerID="cri-o://525890e6f23eed865f7ea6ad9cde66c3fc22211122e1a3a1c5636f249c99399e" gracePeriod=30
Sep 18 20:39:56.175090 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:56.175041    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kube-rbac-proxy" containerID="cri-o://55ae256761d304663e58d07832eaa4b70ceb6e0c1b6aa4f3caa48c27090c004c" gracePeriod=30
Sep 18 20:39:56.203481 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:56.203439    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:39:57.174501 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:57.174473    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:39:57.174935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:57.174904    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:39:57.196109 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:57.196079    2557 generic.go:356] "Generic (PLEG): container finished" podID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerID="55ae256761d304663e58d07832eaa4b70ceb6e0c1b6aa4f3caa48c27090c004c" exitCode=2
Sep 18 20:39:57.196243 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:57.196150    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" event={"ID":"ade359b5-6135-45a0-b967-0690b6b9fbbc","Type":"ContainerDied","Data":"55ae256761d304663e58d07832eaa4b70ceb6e0c1b6aa4f3caa48c27090c004c"}
Sep 18 20:39:57.795641 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:57.795603    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 18 20:39:57.800697 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:39:57.800670    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:40:00.212258 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.212229    2557 generic.go:356] "Generic (PLEG): container finished" podID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerID="525890e6f23eed865f7ea6ad9cde66c3fc22211122e1a3a1c5636f249c99399e" exitCode=0
Sep 18 20:40:00.212614 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.212303    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" event={"ID":"ade359b5-6135-45a0-b967-0690b6b9fbbc","Type":"ContainerDied","Data":"525890e6f23eed865f7ea6ad9cde66c3fc22211122e1a3a1c5636f249c99399e"}
Sep 18 20:40:00.321772 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.321750    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:40:00.360771 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.360747    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-rbac-proxy-sar-config\") pod \"ade359b5-6135-45a0-b967-0690b6b9fbbc\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") "
Sep 18 20:40:00.360898 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.360783    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q67m\" (UniqueName: \"kubernetes.io/projected/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-api-access-4q67m\") pod \"ade359b5-6135-45a0-b967-0690b6b9fbbc\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") "
Sep 18 20:40:00.360898 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.360808    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade359b5-6135-45a0-b967-0690b6b9fbbc-proxy-tls\") pod \"ade359b5-6135-45a0-b967-0690b6b9fbbc\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") "
Sep 18 20:40:00.360898 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.360851    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade359b5-6135-45a0-b967-0690b6b9fbbc-kserve-provision-location\") pod \"ade359b5-6135-45a0-b967-0690b6b9fbbc\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") "
Sep 18 20:40:00.361066 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.360908    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-cabundle-cert\") pod \"ade359b5-6135-45a0-b967-0690b6b9fbbc\" (UID: \"ade359b5-6135-45a0-b967-0690b6b9fbbc\") "
Sep 18 20:40:00.361203 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.361170    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ade359b5-6135-45a0-b967-0690b6b9fbbc" (UID: "ade359b5-6135-45a0-b967-0690b6b9fbbc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:40:00.361309 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.361226    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade359b5-6135-45a0-b967-0690b6b9fbbc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ade359b5-6135-45a0-b967-0690b6b9fbbc" (UID: "ade359b5-6135-45a0-b967-0690b6b9fbbc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:40:00.361309 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.361286    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ade359b5-6135-45a0-b967-0690b6b9fbbc" (UID: "ade359b5-6135-45a0-b967-0690b6b9fbbc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:40:00.362842 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.362815    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade359b5-6135-45a0-b967-0690b6b9fbbc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ade359b5-6135-45a0-b967-0690b6b9fbbc" (UID: "ade359b5-6135-45a0-b967-0690b6b9fbbc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:40:00.362929 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.362906    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-api-access-4q67m" (OuterVolumeSpecName: "kube-api-access-4q67m") pod "ade359b5-6135-45a0-b967-0690b6b9fbbc" (UID: "ade359b5-6135-45a0-b967-0690b6b9fbbc"). InnerVolumeSpecName "kube-api-access-4q67m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:40:00.462026 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.462002    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade359b5-6135-45a0-b967-0690b6b9fbbc-kserve-provision-location\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:40:00.462026 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.462021    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-cabundle-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:40:00.462154 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.462031    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:40:00.462154 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.462040    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q67m\" (UniqueName: \"kubernetes.io/projected/ade359b5-6135-45a0-b967-0690b6b9fbbc-kube-api-access-4q67m\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:40:00.462154 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:00.462048    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade359b5-6135-45a0-b967-0690b6b9fbbc-proxy-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:40:01.219303 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:01.219265    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp" event={"ID":"ade359b5-6135-45a0-b967-0690b6b9fbbc","Type":"ContainerDied","Data":"e45ef6d22405f1383e7cc3fb24f16053966cd4e25e774800f08a0829fd826ee4"}
Sep 18 20:40:01.219763 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:01.219307    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"
Sep 18 20:40:01.219763 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:01.219346    2557 scope.go:117] "RemoveContainer" containerID="55ae256761d304663e58d07832eaa4b70ceb6e0c1b6aa4f3caa48c27090c004c"
Sep 18 20:40:01.230980 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:01.230959    2557 scope.go:117] "RemoveContainer" containerID="525890e6f23eed865f7ea6ad9cde66c3fc22211122e1a3a1c5636f249c99399e"
Sep 18 20:40:01.236680 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:01.236658    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"]
Sep 18 20:40:01.240756 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:01.240733    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-89cb6-predictor-8c65998fd-nq5qp"]
Sep 18 20:40:01.240873 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:01.240826    2557 scope.go:117] "RemoveContainer" containerID="1d7bd584e5ee3050d248675efa6bc3c9f3803b9d26180db421eb4d56dd947383"
Sep 18 20:40:02.723036 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:02.723008    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" path="/var/lib/kubelet/pods/ade359b5-6135-45a0-b967-0690b6b9fbbc/volumes"
Sep 18 20:40:07.175833 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:07.175793    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:40:17.175217 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:17.175175    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:40:27.175598 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:27.175555    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:40:37.175889 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:37.175849    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:40:38.699521 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:38.699496    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:40:38.701776 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:38.701755    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:40:43.230776 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:43.230728    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:40:47.175519 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:47.175479    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:40:57.175596 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:40:57.175555    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:41:07.176669 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:07.176637    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:41:14.021742 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:14.021700    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"]
Sep 18 20:41:14.022158 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:14.022028    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" containerID="cri-o://2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498" gracePeriod=30
Sep 18 20:41:14.022158 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:14.022070    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kube-rbac-proxy" containerID="cri-o://c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884" gracePeriod=30
Sep 18 20:41:14.046958 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:14.046931    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:14.553968 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:14.553932    2557 generic.go:356] "Generic (PLEG): container finished" podID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerID="c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884" exitCode=2
Sep 18 20:41:14.554108 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:14.553996    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" event={"ID":"4df53336-693f-47d2-ae93-7eabcb1a416c","Type":"ContainerDied","Data":"c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884"}
Sep 18 20:41:15.241165 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:15.241130    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:17.170933 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.170895    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 18 20:41:17.175170 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.175144    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:41:17.965567 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.965545    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:41:17.990536 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.990510    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-cabundle-cert\") pod \"4df53336-693f-47d2-ae93-7eabcb1a416c\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") "
Sep 18 20:41:17.990658 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.990558    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df53336-693f-47d2-ae93-7eabcb1a416c-proxy-tls\") pod \"4df53336-693f-47d2-ae93-7eabcb1a416c\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") "
Sep 18 20:41:17.990658 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.990591    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-rbac-proxy-sar-config\") pod \"4df53336-693f-47d2-ae93-7eabcb1a416c\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") "
Sep 18 20:41:17.990658 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.990642    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df53336-693f-47d2-ae93-7eabcb1a416c-kserve-provision-location\") pod \"4df53336-693f-47d2-ae93-7eabcb1a416c\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") "
Sep 18 20:41:17.990829 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.990676    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btwv9\" (UniqueName: \"kubernetes.io/projected/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-api-access-btwv9\") pod \"4df53336-693f-47d2-ae93-7eabcb1a416c\" (UID: \"4df53336-693f-47d2-ae93-7eabcb1a416c\") "
Sep 18 20:41:17.990948 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.990840    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4df53336-693f-47d2-ae93-7eabcb1a416c" (UID: "4df53336-693f-47d2-ae93-7eabcb1a416c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:41:17.991035 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.990987    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df53336-693f-47d2-ae93-7eabcb1a416c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4df53336-693f-47d2-ae93-7eabcb1a416c" (UID: "4df53336-693f-47d2-ae93-7eabcb1a416c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:17.991086 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.991025    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4df53336-693f-47d2-ae93-7eabcb1a416c" (UID: "4df53336-693f-47d2-ae93-7eabcb1a416c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:41:17.992664 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.992642    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df53336-693f-47d2-ae93-7eabcb1a416c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4df53336-693f-47d2-ae93-7eabcb1a416c" (UID: "4df53336-693f-47d2-ae93-7eabcb1a416c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:41:17.992776 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:17.992705    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-api-access-btwv9" (OuterVolumeSpecName: "kube-api-access-btwv9") pod "4df53336-693f-47d2-ae93-7eabcb1a416c" (UID: "4df53336-693f-47d2-ae93-7eabcb1a416c"). InnerVolumeSpecName "kube-api-access-btwv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:41:18.091336 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.091287    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-cabundle-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:41:18.091336 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.091306    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df53336-693f-47d2-ae93-7eabcb1a416c-proxy-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:41:18.091336 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.091315    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:41:18.091336 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.091324    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df53336-693f-47d2-ae93-7eabcb1a416c-kserve-provision-location\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:41:18.091336 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.091334    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btwv9\" (UniqueName: \"kubernetes.io/projected/4df53336-693f-47d2-ae93-7eabcb1a416c-kube-api-access-btwv9\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:41:18.574678 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.574646    2557 generic.go:356] "Generic (PLEG): container finished" podID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerID="2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498" exitCode=0
Sep 18 20:41:18.575108 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.574701    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"
Sep 18 20:41:18.575108 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.574731    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" event={"ID":"4df53336-693f-47d2-ae93-7eabcb1a416c","Type":"ContainerDied","Data":"2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498"}
Sep 18 20:41:18.575108 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.574811    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx" event={"ID":"4df53336-693f-47d2-ae93-7eabcb1a416c","Type":"ContainerDied","Data":"466a2e4a322b08454511f565eb4b5a8246262b203a470f9c1f36731f5c90b514"}
Sep 18 20:41:18.575108 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.574850    2557 scope.go:117] "RemoveContainer" containerID="c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884"
Sep 18 20:41:18.583893 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.583865    2557 scope.go:117] "RemoveContainer" containerID="2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498"
Sep 18 20:41:18.591911 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.591895    2557 scope.go:117] "RemoveContainer" containerID="ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8"
Sep 18 20:41:18.599500 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.599487    2557 scope.go:117] "RemoveContainer" containerID="c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884"
Sep 18 20:41:18.599777 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:41:18.599758    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884\": container with ID starting with c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884 not found: ID does not exist" containerID="c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884"
Sep 18 20:41:18.599840 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.599785    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884"} err="failed to get container status \"c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884\": rpc error: code = NotFound desc = could not find container \"c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884\": container with ID starting with c81eaa0f2a49c1987517cd846af9f9cc6a66b7f2b6724c357d2c2251034a4884 not found: ID does not exist"
Sep 18 20:41:18.599840 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.599803    2557 scope.go:117] "RemoveContainer" containerID="2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498"
Sep 18 20:41:18.600069 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:41:18.600050    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498\": container with ID starting with 2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498 not found: ID does not exist" containerID="2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498"
Sep 18 20:41:18.600187 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.600074    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498"} err="failed to get container status \"2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498\": rpc error: code = NotFound desc = could not find container \"2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498\": container with ID starting with 2b69c4dfd66193253d99dbe39f2203c68615f4b610ea206919eeba41b81d3498 not found: ID does not exist"
Sep 18 20:41:18.600187 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.600090    2557 scope.go:117] "RemoveContainer" containerID="ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8"
Sep 18 20:41:18.601072 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:41:18.601031    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8\": container with ID starting with ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8 not found: ID does not exist" containerID="ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8"
Sep 18 20:41:18.601171 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.601066    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8"} err="failed to get container status \"ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8\": rpc error: code = NotFound desc = could not find container \"ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8\": container with ID starting with ab884c0c7a9189e7177cad4a0cdcdf563369c74b090c6b98eee506f823c622c8 not found: ID does not exist"
Sep 18 20:41:18.602827 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.602806    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"]
Sep 18 20:41:18.605369 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.605351    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2f5vx"]
Sep 18 20:41:18.723060 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:18.723034    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" path="/var/lib/kubelet/pods/4df53336-693f-47d2-ae93-7eabcb1a416c/volumes"
Sep 18 20:41:19.141582 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.141497    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:19.254033 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.254000    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:19.270390 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270361    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"]
Sep 18 20:41:19.270793 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270777    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kube-rbac-proxy"
Sep 18 20:41:19.270880 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270796    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kube-rbac-proxy"
Sep 18 20:41:19.270880 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270811    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container"
Sep 18 20:41:19.270880 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270820    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container"
Sep 18 20:41:19.270880 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270849    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="storage-initializer"
Sep 18 20:41:19.270880 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270857    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="storage-initializer"
Sep 18 20:41:19.270880 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270867    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kube-rbac-proxy"
Sep 18 20:41:19.270880 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270875    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kube-rbac-proxy"
Sep 18 20:41:19.271239 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270887    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="storage-initializer"
Sep 18 20:41:19.271239 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270896    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="storage-initializer"
Sep 18 20:41:19.271239 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270913    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container"
Sep 18 20:41:19.271239 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.270921    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container"
Sep 18 20:41:19.271239 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.271001    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kube-rbac-proxy"
Sep 18 20:41:19.271239 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.271015    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kube-rbac-proxy"
Sep 18 20:41:19.271239 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.271026    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df53336-693f-47d2-ae93-7eabcb1a416c" containerName="kserve-container"
Sep 18 20:41:19.271239 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.271035    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade359b5-6135-45a0-b967-0690b6b9fbbc" containerName="kserve-container"
Sep 18 20:41:19.275754 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.275732    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.284569 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.284550    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ttgfx\""
Sep 18 20:41:19.287752 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.287734    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:41:19.287837 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.287734    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 20:41:19.288671 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.288655    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 20:41:19.288720 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.288689    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:41:19.297405 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.297389    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:41:19.297670 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.297648    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:41:19.298750 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.298734    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"]
Sep 18 20:41:19.403389 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.403325    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.403389 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.403353    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31035aaa-3d2d-4497-a224-8cc3ee63f708-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.403389 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.403377    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxj9s\" (UniqueName: \"kubernetes.io/projected/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-api-access-mxj9s\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.403600 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.403405    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31035aaa-3d2d-4497-a224-8cc3ee63f708-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.403600 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.403435    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.405394 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.405368    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:19.425209 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.425185    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"]
Sep 18 20:41:19.428641 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.428627    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.431129 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.431112    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 20:41:19.447106 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.447085    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"]
Sep 18 20:41:19.503177 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.503147    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:19.504169 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504150    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.504233 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504179    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31035aaa-3d2d-4497-a224-8cc3ee63f708-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.504233 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504207    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxj9s\" (UniqueName: \"kubernetes.io/projected/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-api-access-mxj9s\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.504306 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504232    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31035aaa-3d2d-4497-a224-8cc3ee63f708-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.504306 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504256    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.504306 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504281    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/653bfd3e-0222-48a6-bf5c-26d34825a1d7-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.504446 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.504446 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnf9p\" (UniqueName: \"kubernetes.io/projected/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-api-access-nnf9p\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.504578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.504631 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.504737 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.504720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31035aaa-3d2d-4497-a224-8cc3ee63f708-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.505099 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.505075    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.505099 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.505090    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.506832 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.506810    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31035aaa-3d2d-4497-a224-8cc3ee63f708-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.512598 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.512575    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxj9s\" (UniqueName: \"kubernetes.io/projected/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-api-access-mxj9s\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.585800 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.585777    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:19.604921 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.604896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/653bfd3e-0222-48a6-bf5c-26d34825a1d7-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.605022 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.604934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.605022 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.604958    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnf9p\" (UniqueName: \"kubernetes.io/projected/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-api-access-nnf9p\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.605022 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.604983    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.605185 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.605020    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.605444 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.605413    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.605742 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.605721    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.605931 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.605721    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.607694 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.607674    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/653bfd3e-0222-48a6-bf5c-26d34825a1d7-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.613168 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.613147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnf9p\" (UniqueName: \"kubernetes.io/projected/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-api-access-nnf9p\") pod \"isvc-canary-promote-predictor-6546f8697f-8rslz\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.738181 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.738147    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:19.859687 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:19.859505    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"]
Sep 18 20:41:19.861297 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:41:19.861272    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31035aaa_3d2d_4497_a224_8cc3ee63f708.slice/crio-bbac96e4231d5f7aa664ee0ec166ea53617e9f932cf892f2563a03c7500afce1 WatchSource:0}: Error finding container bbac96e4231d5f7aa664ee0ec166ea53617e9f932cf892f2563a03c7500afce1: Status 404 returned error can't find the container with id bbac96e4231d5f7aa664ee0ec166ea53617e9f932cf892f2563a03c7500afce1
Sep 18 20:41:20.012997 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:20.012913    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"]
Sep 18 20:41:20.015059 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:41:20.015033    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653bfd3e_0222_48a6_bf5c_26d34825a1d7.slice/crio-247b22c261ac7a5e433c0083d984888f3e2120d3483c7464f971eb73a7c9db09 WatchSource:0}: Error finding container 247b22c261ac7a5e433c0083d984888f3e2120d3483c7464f971eb73a7c9db09: Status 404 returned error can't find the container with id 247b22c261ac7a5e433c0083d984888f3e2120d3483c7464f971eb73a7c9db09
Sep 18 20:41:20.586447 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:20.586411    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" event={"ID":"31035aaa-3d2d-4497-a224-8cc3ee63f708","Type":"ContainerStarted","Data":"7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7"}
Sep 18 20:41:20.586904 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:20.586492    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" event={"ID":"31035aaa-3d2d-4497-a224-8cc3ee63f708","Type":"ContainerStarted","Data":"bbac96e4231d5f7aa664ee0ec166ea53617e9f932cf892f2563a03c7500afce1"}
Sep 18 20:41:20.588009 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:20.587975    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" event={"ID":"653bfd3e-0222-48a6-bf5c-26d34825a1d7","Type":"ContainerStarted","Data":"e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c"}
Sep 18 20:41:20.588132 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:20.588036    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" event={"ID":"653bfd3e-0222-48a6-bf5c-26d34825a1d7","Type":"ContainerStarted","Data":"247b22c261ac7a5e433c0083d984888f3e2120d3483c7464f971eb73a7c9db09"}
Sep 18 20:41:24.605502 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:24.605470    2557 generic.go:356] "Generic (PLEG): container finished" podID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerID="7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7" exitCode=0
Sep 18 20:41:24.605917 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:24.605524    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" event={"ID":"31035aaa-3d2d-4497-a224-8cc3ee63f708","Type":"ContainerDied","Data":"7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7"}
Sep 18 20:41:24.607129 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:24.607108    2557 generic.go:356] "Generic (PLEG): container finished" podID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerID="e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c" exitCode=0
Sep 18 20:41:24.607189 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:24.607167    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" event={"ID":"653bfd3e-0222-48a6-bf5c-26d34825a1d7","Type":"ContainerDied","Data":"e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c"}
Sep 18 20:41:25.613999 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.613961    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" event={"ID":"31035aaa-3d2d-4497-a224-8cc3ee63f708","Type":"ContainerStarted","Data":"9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6"}
Sep 18 20:41:25.614385 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.614028    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" event={"ID":"31035aaa-3d2d-4497-a224-8cc3ee63f708","Type":"ContainerStarted","Data":"03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda"}
Sep 18 20:41:25.614385 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.614337    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:25.614548 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.614400    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:25.615994 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.615962    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:41:25.616490 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.616467    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" event={"ID":"653bfd3e-0222-48a6-bf5c-26d34825a1d7","Type":"ContainerStarted","Data":"0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a"}
Sep 18 20:41:25.616583 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.616517    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" event={"ID":"653bfd3e-0222-48a6-bf5c-26d34825a1d7","Type":"ContainerStarted","Data":"e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc"}
Sep 18 20:41:25.616853 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.616835    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:25.616908 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.616889    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:25.617893 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.617868    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:41:25.650878 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.650824    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podStartSLOduration=2.619773083 podStartE2EDuration="6.650806308s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.031033225s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:41:25.647518138 +0000 UTC m=+1249.140870716" watchObservedRunningTime="2026-09-18 20:41:25.650806308 +0000 UTC m=+1249.144158887"
Sep 18 20:41:25.675165 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:25.675126    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podStartSLOduration=2.724120458 podStartE2EDuration="6.675115306s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.950994848s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:41:25.67481122 +0000 UTC m=+1249.168163798" watchObservedRunningTime="2026-09-18 20:41:25.675115306 +0000 UTC m=+1249.168467884"
Sep 18 20:41:26.621242 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:26.621208    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:41:26.621631 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:26.621352    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:41:31.625503 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:31.625466    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:41:31.625935 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:31.625883    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:41:31.625978 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:31.625950    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:41:31.626374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:31.626351    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:41:41.626560 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:41.626517    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:41:41.626946 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:41.626517    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:41:51.626653 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:51.626616    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:41:51.627053 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:41:51.626617    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:42:01.626359 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:01.626315    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:42:01.626876 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:01.626315    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:42:11.626478 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:11.626419    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:42:11.626478 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:11.626437    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:42:21.626079 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:21.626040    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:42:21.626584 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:21.626306    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:42:31.626090 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:31.626041    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:42:31.626591 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:31.626255    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:42:38.723809 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:38.723780    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:42:41.627283 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:41.627250    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:42:49.307953 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:49.307858    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"]
Sep 18 20:42:49.308440 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:49.308146    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container" containerID="cri-o://e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc" gracePeriod=30
Sep 18 20:42:49.308440 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:49.308169    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kube-rbac-proxy" containerID="cri-o://0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a" gracePeriod=30
Sep 18 20:42:49.318348 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:49.318314    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:42:50.015064 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:50.015029    2557 generic.go:356] "Generic (PLEG): container finished" podID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerID="0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a" exitCode=2
Sep 18 20:42:50.015215 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:50.015078    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" event={"ID":"653bfd3e-0222-48a6-bf5c-26d34825a1d7","Type":"ContainerDied","Data":"0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a"}
Sep 18 20:42:51.622173 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:51.622121    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 18 20:42:53.350405 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.350382    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:42:53.421566 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.421508    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-rbac-proxy-sar-config\") pod \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") "
Sep 18 20:42:53.421566 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.421563    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-cabundle-cert\") pod \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") "
Sep 18 20:42:53.421716 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.421597    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kserve-provision-location\") pod \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") "
Sep 18 20:42:53.421716 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.421635    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/653bfd3e-0222-48a6-bf5c-26d34825a1d7-proxy-tls\") pod \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") "
Sep 18 20:42:53.421716 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.421655    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnf9p\" (UniqueName: \"kubernetes.io/projected/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-api-access-nnf9p\") pod \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\" (UID: \"653bfd3e-0222-48a6-bf5c-26d34825a1d7\") "
Sep 18 20:42:53.421924 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.421899    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "653bfd3e-0222-48a6-bf5c-26d34825a1d7" (UID: "653bfd3e-0222-48a6-bf5c-26d34825a1d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:42:53.421995 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.421904    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "653bfd3e-0222-48a6-bf5c-26d34825a1d7" (UID: "653bfd3e-0222-48a6-bf5c-26d34825a1d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:42:53.421995 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.421936    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "653bfd3e-0222-48a6-bf5c-26d34825a1d7" (UID: "653bfd3e-0222-48a6-bf5c-26d34825a1d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:42:53.423721 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.423698    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-api-access-nnf9p" (OuterVolumeSpecName: "kube-api-access-nnf9p") pod "653bfd3e-0222-48a6-bf5c-26d34825a1d7" (UID: "653bfd3e-0222-48a6-bf5c-26d34825a1d7"). InnerVolumeSpecName "kube-api-access-nnf9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:42:53.423795 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.423770    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653bfd3e-0222-48a6-bf5c-26d34825a1d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "653bfd3e-0222-48a6-bf5c-26d34825a1d7" (UID: "653bfd3e-0222-48a6-bf5c-26d34825a1d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:42:53.522356 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.522322    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:42:53.522356 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.522352    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/653bfd3e-0222-48a6-bf5c-26d34825a1d7-cabundle-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:42:53.522356 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.522363    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kserve-provision-location\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:42:53.522550 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.522372    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/653bfd3e-0222-48a6-bf5c-26d34825a1d7-proxy-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:42:53.522550 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:53.522381    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnf9p\" (UniqueName: \"kubernetes.io/projected/653bfd3e-0222-48a6-bf5c-26d34825a1d7-kube-api-access-nnf9p\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:42:54.035374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.035343    2557 generic.go:356] "Generic (PLEG): container finished" podID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerID="e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc" exitCode=0
Sep 18 20:42:54.035584 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.035402    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"
Sep 18 20:42:54.035584 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.035428    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" event={"ID":"653bfd3e-0222-48a6-bf5c-26d34825a1d7","Type":"ContainerDied","Data":"e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc"}
Sep 18 20:42:54.035584 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.035521    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz" event={"ID":"653bfd3e-0222-48a6-bf5c-26d34825a1d7","Type":"ContainerDied","Data":"247b22c261ac7a5e433c0083d984888f3e2120d3483c7464f971eb73a7c9db09"}
Sep 18 20:42:54.035584 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.035560    2557 scope.go:117] "RemoveContainer" containerID="0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a"
Sep 18 20:42:54.044420 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.044407    2557 scope.go:117] "RemoveContainer" containerID="e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc"
Sep 18 20:42:54.051970 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.051955    2557 scope.go:117] "RemoveContainer" containerID="e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c"
Sep 18 20:42:54.061643 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.061625    2557 scope.go:117] "RemoveContainer" containerID="0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a"
Sep 18 20:42:54.061928 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:42:54.061910    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a\": container with ID starting with 0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a not found: ID does not exist" containerID="0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a"
Sep 18 20:42:54.062019 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.061938    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a"} err="failed to get container status \"0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a\": rpc error: code = NotFound desc = could not find container \"0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a\": container with ID starting with 0609098b1c3d762ab52da2ac659f9112343f0e651ff060fe1ce8f247fe104c9a not found: ID does not exist"
Sep 18 20:42:54.062019 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.061954    2557 scope.go:117] "RemoveContainer" containerID="e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc"
Sep 18 20:42:54.062330 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:42:54.062308    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc\": container with ID starting with e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc not found: ID does not exist" containerID="e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc"
Sep 18 20:42:54.062437 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.062337    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc"} err="failed to get container status \"e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc\": rpc error: code = NotFound desc = could not find container \"e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc\": container with ID starting with e694a6135668191dc999fe3993b7e33abc66a48642a96d425b4231cab6a71fdc not found: ID does not exist"
Sep 18 20:42:54.062437 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.062357    2557 scope.go:117] "RemoveContainer" containerID="e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c"
Sep 18 20:42:54.062787 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:42:54.062719    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c\": container with ID starting with e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c not found: ID does not exist" containerID="e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c"
Sep 18 20:42:54.062787 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.062760    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c"} err="failed to get container status \"e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c\": rpc error: code = NotFound desc = could not find container \"e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c\": container with ID starting with e9e2ad9d6fa46f54d8d22180f11b1722aae2752e1ac5fb4263383f45582bc95c not found: ID does not exist"
Sep 18 20:42:54.063611 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.063587    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"]
Sep 18 20:42:54.065132 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.065111    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-8rslz"]
Sep 18 20:42:54.722310 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:42:54.722283    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" path="/var/lib/kubelet/pods/653bfd3e-0222-48a6-bf5c-26d34825a1d7/volumes"
Sep 18 20:43:28.215355 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:43:28.215317    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:19.505281 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:19.505194    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"]
Sep 18 20:44:19.505889 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:19.505533    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" containerID="cri-o://03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda" gracePeriod=30
Sep 18 20:44:19.505889 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:19.505559    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kube-rbac-proxy" containerID="cri-o://9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6" gracePeriod=30
Sep 18 20:44:19.538711 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:19.538679    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:20.430599 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:20.430560    2557 generic.go:356] "Generic (PLEG): container finished" podID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerID="9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6" exitCode=2
Sep 18 20:44:20.430777 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:20.430630    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" event={"ID":"31035aaa-3d2d-4497-a224-8cc3ee63f708","Type":"ContainerDied","Data":"9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6"}
Sep 18 20:44:21.622426 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:21.622379    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 20:44:21.626734 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:21.626703    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:44:24.651414 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.651386    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:44:24.653421 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653399    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31035aaa-3d2d-4497-a224-8cc3ee63f708-kserve-provision-location\") pod \"31035aaa-3d2d-4497-a224-8cc3ee63f708\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") "
Sep 18 20:44:24.653551 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653464    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-rbac-proxy-sar-config\") pod \"31035aaa-3d2d-4497-a224-8cc3ee63f708\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") "
Sep 18 20:44:24.653551 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653490    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-cabundle-cert\") pod \"31035aaa-3d2d-4497-a224-8cc3ee63f708\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") "
Sep 18 20:44:24.653672 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653561    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxj9s\" (UniqueName: \"kubernetes.io/projected/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-api-access-mxj9s\") pod \"31035aaa-3d2d-4497-a224-8cc3ee63f708\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") "
Sep 18 20:44:24.653672 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653591    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31035aaa-3d2d-4497-a224-8cc3ee63f708-proxy-tls\") pod \"31035aaa-3d2d-4497-a224-8cc3ee63f708\" (UID: \"31035aaa-3d2d-4497-a224-8cc3ee63f708\") "
Sep 18 20:44:24.653768 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653718    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31035aaa-3d2d-4497-a224-8cc3ee63f708-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31035aaa-3d2d-4497-a224-8cc3ee63f708" (UID: "31035aaa-3d2d-4497-a224-8cc3ee63f708"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:24.653847 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653826    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31035aaa-3d2d-4497-a224-8cc3ee63f708" (UID: "31035aaa-3d2d-4497-a224-8cc3ee63f708"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:44:24.653891 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653867    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31035aaa-3d2d-4497-a224-8cc3ee63f708" (UID: "31035aaa-3d2d-4497-a224-8cc3ee63f708"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:44:24.653947 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653932    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31035aaa-3d2d-4497-a224-8cc3ee63f708-kserve-provision-location\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:44:24.653987 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653951    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:44:24.653987 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.653959    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31035aaa-3d2d-4497-a224-8cc3ee63f708-cabundle-cert\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:44:24.655698 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.655671    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31035aaa-3d2d-4497-a224-8cc3ee63f708-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31035aaa-3d2d-4497-a224-8cc3ee63f708" (UID: "31035aaa-3d2d-4497-a224-8cc3ee63f708"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:44:24.655901 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.655880    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-api-access-mxj9s" (OuterVolumeSpecName: "kube-api-access-mxj9s") pod "31035aaa-3d2d-4497-a224-8cc3ee63f708" (UID: "31035aaa-3d2d-4497-a224-8cc3ee63f708"). InnerVolumeSpecName "kube-api-access-mxj9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:44:24.754366 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.754341    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxj9s\" (UniqueName: \"kubernetes.io/projected/31035aaa-3d2d-4497-a224-8cc3ee63f708-kube-api-access-mxj9s\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:44:24.754366 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:24.754366    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31035aaa-3d2d-4497-a224-8cc3ee63f708-proxy-tls\") on node \"ip-10-0-138-28.ec2.internal\" DevicePath \"\""
Sep 18 20:44:25.454415 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.454382    2557 generic.go:356] "Generic (PLEG): container finished" podID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerID="03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda" exitCode=0
Sep 18 20:44:25.454612 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.454438    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"
Sep 18 20:44:25.454612 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.454437    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" event={"ID":"31035aaa-3d2d-4497-a224-8cc3ee63f708","Type":"ContainerDied","Data":"03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda"}
Sep 18 20:44:25.454612 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.454586    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz" event={"ID":"31035aaa-3d2d-4497-a224-8cc3ee63f708","Type":"ContainerDied","Data":"bbac96e4231d5f7aa664ee0ec166ea53617e9f932cf892f2563a03c7500afce1"}
Sep 18 20:44:25.454726 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.454618    2557 scope.go:117] "RemoveContainer" containerID="9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6"
Sep 18 20:44:25.465522 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.465444    2557 scope.go:117] "RemoveContainer" containerID="03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda"
Sep 18 20:44:25.472296 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.472271    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"]
Sep 18 20:44:25.473895 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.473877    2557 scope.go:117] "RemoveContainer" containerID="7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7"
Sep 18 20:44:25.476723 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.476700    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-688sz"]
Sep 18 20:44:25.481774 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.481754    2557 scope.go:117] "RemoveContainer" containerID="9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6"
Sep 18 20:44:25.482042 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:44:25.482022    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6\": container with ID starting with 9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6 not found: ID does not exist" containerID="9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6"
Sep 18 20:44:25.482088 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.482051    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6"} err="failed to get container status \"9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6\": rpc error: code = NotFound desc = could not find container \"9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6\": container with ID starting with 9f5ebae6d4518b6b840673e2c0bccdc978af985b3ad960278acf592c8e1ff8f6 not found: ID does not exist"
Sep 18 20:44:25.482088 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.482069    2557 scope.go:117] "RemoveContainer" containerID="03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda"
Sep 18 20:44:25.482342 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:44:25.482321    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda\": container with ID starting with 03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda not found: ID does not exist" containerID="03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda"
Sep 18 20:44:25.482396 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.482348    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda"} err="failed to get container status \"03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda\": rpc error: code = NotFound desc = could not find container \"03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda\": container with ID starting with 03585233d8d6bed62eb35c2eb11480575c1fc2fd800c98156cb79667ee5d3dda not found: ID does not exist"
Sep 18 20:44:25.482396 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.482365    2557 scope.go:117] "RemoveContainer" containerID="7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7"
Sep 18 20:44:25.482590 ip-10-0-138-28 kubenswrapper[2557]: E0918 20:44:25.482575    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7\": container with ID starting with 7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7 not found: ID does not exist" containerID="7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7"
Sep 18 20:44:25.482635 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:25.482592    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7"} err="failed to get container status \"7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7\": rpc error: code = NotFound desc = could not find container \"7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7\": container with ID starting with 7666859f5139897c261c357eb0046c886cc8eb70f87ae21738ac07faf184ffc7 not found: ID does not exist"
Sep 18 20:44:26.723819 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:26.723779    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" path="/var/lib/kubelet/pods/31035aaa-3d2d-4497-a224-8cc3ee63f708/volumes"
Sep 18 20:44:33.958630 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:33.958592    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:34.036830 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:34.036790    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:54.632855 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:54.632812    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:54.733998 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:44:54.733947    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:45:38.736370 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:45:38.736343    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:45:38.739993 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:45:38.739973    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:46:29.751442 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:46:29.751399    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:46:30.969752 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:46:30.969718    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:46:31.071304 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:46:31.071270    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:46:31.171675 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:46:31.171647    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:46:49.808957 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:46:49.808900    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:46:49.863814 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:46:49.863784    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:01.154097 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:01.154056    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:01.301675 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:01.301641    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:01.374763 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:01.374728    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:04.251907 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:04.251878    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l6x2p_62a9d73d-1877-48b9-94e2-5a76311b7453/global-pull-secret-syncer/0.log"
Sep 18 20:47:04.298466 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:04.298418    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log"
Sep 18 20:47:04.434330 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:04.434298    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-28.ec2.internal_7d826f4c7a124136216df58c6a6c0d1b/haproxy/0.log"
Sep 18 20:47:06.816377 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:06.816333    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:06.892666 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:06.892625    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n4b_1fa00627-119a-42c6-aff3-568d9234c049/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:08.495189 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.495164    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58cea09a-cf80-4b94-83df-7a1487dfd5aa/alertmanager/0.log"
Sep 18 20:47:08.519128 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.519097    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58cea09a-cf80-4b94-83df-7a1487dfd5aa/config-reloader/0.log"
Sep 18 20:47:08.550354 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.550325    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58cea09a-cf80-4b94-83df-7a1487dfd5aa/kube-rbac-proxy-web/0.log"
Sep 18 20:47:08.581100 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.581074    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58cea09a-cf80-4b94-83df-7a1487dfd5aa/kube-rbac-proxy/0.log"
Sep 18 20:47:08.611322 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.611296    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58cea09a-cf80-4b94-83df-7a1487dfd5aa/kube-rbac-proxy-metric/0.log"
Sep 18 20:47:08.639119 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.639088    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58cea09a-cf80-4b94-83df-7a1487dfd5aa/prom-label-proxy/0.log"
Sep 18 20:47:08.665352 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.665325    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58cea09a-cf80-4b94-83df-7a1487dfd5aa/init-config-reloader/0.log"
Sep 18 20:47:08.732805 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.732778    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-7k4q5_6a6ceca8-214e-43d1-8201-7932354a5ade/kube-state-metrics/0.log"
Sep 18 20:47:08.757931 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.757844    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-7k4q5_6a6ceca8-214e-43d1-8201-7932354a5ade/kube-rbac-proxy-main/0.log"
Sep 18 20:47:08.783727 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.783664    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-7k4q5_6a6ceca8-214e-43d1-8201-7932354a5ade/kube-rbac-proxy-self/0.log"
Sep 18 20:47:08.815912 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.815884    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-554d6d74df-lmtzk_3859bc03-5295-4b0f-90f7-f0d1a8b2df5f/metrics-server/0.log"
Sep 18 20:47:08.840812 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:08.840783    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-qzxjb_a488e4e5-0b3b-4e74-b4c4-38d29c1243f4/monitoring-plugin/0.log"
Sep 18 20:47:09.039711 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:09.039637    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xgrh8_dc94692c-fefc-48ff-a8a7-c7938555cd32/node-exporter/0.log"
Sep 18 20:47:09.064026 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:09.063996    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xgrh8_dc94692c-fefc-48ff-a8a7-c7938555cd32/kube-rbac-proxy/0.log"
Sep 18 20:47:09.086513 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:09.086487    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xgrh8_dc94692c-fefc-48ff-a8a7-c7938555cd32/init-textfile/0.log"
Sep 18 20:47:09.122376 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:09.122347    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-gchwm_0044db96-9a73-4280-adf5-26da2757eb37/kube-rbac-proxy-main/0.log"
Sep 18 20:47:09.146329 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:09.146300    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-gchwm_0044db96-9a73-4280-adf5-26da2757eb37/kube-rbac-proxy-self/0.log"
Sep 18 20:47:09.170558 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:09.170532    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-gchwm_0044db96-9a73-4280-adf5-26da2757eb37/openshift-state-metrics/0.log"
Sep 18 20:47:09.425877 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:09.425796    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-6qxh7_33a265ff-d032-46d3-832e-81d6b02d99a8/prometheus-operator/0.log"
Sep 18 20:47:09.448754 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:09.448726    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-6qxh7_33a265ff-d032-46d3-832e-81d6b02d99a8/kube-rbac-proxy/0.log"
Sep 18 20:47:09.478997 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:09.478970    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-hzclf_7b1188d9-c781-44ce-9fb8-4534f6f1e0e6/prometheus-operator-admission-webhook/0.log"
Sep 18 20:47:12.015289 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.015259    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bb849ff87-zjffq_ce16887f-17ab-41c1-9f4a-65825c10cce8/console/0.log"
Sep 18 20:47:12.048505 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.048470    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-9bhvr_681c9102-7acf-4450-b2b7-2a65aac11518/download-server/0.log"
Sep 18 20:47:12.214162 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214125    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"]
Sep 18 20:47:12.214496 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214481    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="storage-initializer"
Sep 18 20:47:12.214496 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214496    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="storage-initializer"
Sep 18 20:47:12.214585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214505    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kube-rbac-proxy"
Sep 18 20:47:12.214585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214511    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kube-rbac-proxy"
Sep 18 20:47:12.214585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214523    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container"
Sep 18 20:47:12.214585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214529    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container"
Sep 18 20:47:12.214585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214537    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kube-rbac-proxy"
Sep 18 20:47:12.214585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214542    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kube-rbac-proxy"
Sep 18 20:47:12.214585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214554    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="storage-initializer"
Sep 18 20:47:12.214585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214559    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="storage-initializer"
Sep 18 20:47:12.214585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214569    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container"
Sep 18 20:47:12.214585 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214574    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container"
Sep 18 20:47:12.214966 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214624    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kube-rbac-proxy"
Sep 18 20:47:12.214966 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214633    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="653bfd3e-0222-48a6-bf5c-26d34825a1d7" containerName="kserve-container"
Sep 18 20:47:12.214966 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214639    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kserve-container"
Sep 18 20:47:12.214966 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.214645    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="31035aaa-3d2d-4497-a224-8cc3ee63f708" containerName="kube-rbac-proxy"
Sep 18 20:47:12.217815 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.217798    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.219892 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.219871    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dcbdm\"/\"openshift-service-ca.crt\""
Sep 18 20:47:12.220374 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.220357    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dcbdm\"/\"kube-root-ca.crt\""
Sep 18 20:47:12.220426 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.220357    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dcbdm\"/\"default-dockercfg-l6fdb\""
Sep 18 20:47:12.228832 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.228791    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"]
Sep 18 20:47:12.364701 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.364599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-podres\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.364701 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.364666    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-lib-modules\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.364918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.364737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-proc\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.364918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.364790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zrlc\" (UniqueName: \"kubernetes.io/projected/e91555a7-6cf6-43ce-b2a9-da6dcb498285-kube-api-access-9zrlc\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.364918 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.364829    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-sys\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.465960 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.465924    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-sys\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.466144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.465976    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-podres\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.466144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.466015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-lib-modules\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.466144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.466037    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-proc\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.466144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.466063    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrlc\" (UniqueName: \"kubernetes.io/projected/e91555a7-6cf6-43ce-b2a9-da6dcb498285-kube-api-access-9zrlc\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.466144 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.466061    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-sys\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.466372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.466147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-proc\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.466372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.466164    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-podres\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.466372 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.466173    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e91555a7-6cf6-43ce-b2a9-da6dcb498285-lib-modules\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.474153 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.474127    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrlc\" (UniqueName: \"kubernetes.io/projected/e91555a7-6cf6-43ce-b2a9-da6dcb498285-kube-api-access-9zrlc\") pod \"perf-node-gather-daemonset-l6zrt\" (UID: \"e91555a7-6cf6-43ce-b2a9-da6dcb498285\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.528295 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.528256    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:12.803562 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:12.803385    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"]
Sep 18 20:47:12.806240 ip-10-0-138-28 kubenswrapper[2557]: W0918 20:47:12.806200    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode91555a7_6cf6_43ce_b2a9_da6dcb498285.slice/crio-a32869e303bfeead56fd70045e39960f9062d028135b32ce35a97e2547004ca4 WatchSource:0}: Error finding container a32869e303bfeead56fd70045e39960f9062d028135b32ce35a97e2547004ca4: Status 404 returned error can't find the container with id a32869e303bfeead56fd70045e39960f9062d028135b32ce35a97e2547004ca4
Sep 18 20:47:13.234092 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:13.234049    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt" event={"ID":"e91555a7-6cf6-43ce-b2a9-da6dcb498285","Type":"ContainerStarted","Data":"d3e14d1455d05714b8ddea99809da4d19b167005da66ff5aa249f6a628a379af"}
Sep 18 20:47:13.234548 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:13.234146    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt" event={"ID":"e91555a7-6cf6-43ce-b2a9-da6dcb498285","Type":"ContainerStarted","Data":"a32869e303bfeead56fd70045e39960f9062d028135b32ce35a97e2547004ca4"}
Sep 18 20:47:13.234548 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:13.234176    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:13.251402 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:13.251343    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt" podStartSLOduration=1.251325832 podStartE2EDuration="1.251325832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:47:13.250638377 +0000 UTC m=+1596.743990956" watchObservedRunningTime="2026-09-18 20:47:13.251325832 +0000 UTC m=+1596.744678453"
Sep 18 20:47:13.618366 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:13.618291    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j6v8w_d3a5a620-20ab-44e2-a5dd-89b7c7969dcf/dns/0.log"
Sep 18 20:47:13.639470 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:13.639437    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j6v8w_d3a5a620-20ab-44e2-a5dd-89b7c7969dcf/kube-rbac-proxy/0.log"
Sep 18 20:47:13.684875 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:13.684836    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v2ks2_fcceb3e3-7696-404d-a730-edc53df512cc/dns-node-resolver/0.log"
Sep 18 20:47:14.155892 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:14.155860    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7548dd9977-tkk6p_2cd1cf35-e51f-44ec-84fb-d0e22477bbe9/registry/0.log"
Sep 18 20:47:14.202578 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:14.202549    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jnctc_cc34fc3b-3dee-4485-87ba-988209781305/node-ca/0.log"
Sep 18 20:47:15.324442 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:15.324410    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mxhpq_34aed437-eb0e-42d7-94fe-4829714cecc6/serve-healthcheck-canary/0.log"
Sep 18 20:47:15.773654 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:15.773631    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jlz2j_f94eddb8-352a-4eaa-9301-26a28b73b7fe/kube-rbac-proxy/0.log"
Sep 18 20:47:15.798002 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:15.797967    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jlz2j_f94eddb8-352a-4eaa-9301-26a28b73b7fe/exporter/0.log"
Sep 18 20:47:15.822098 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:15.822071    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jlz2j_f94eddb8-352a-4eaa-9301-26a28b73b7fe/extractor/0.log"
Sep 18 20:47:18.274612 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:18.274578    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7dcf4bff6-4g7gd_0abcd90c-ff2c-4f0a-a738-9989825e32f0/manager/0.log"
Sep 18 20:47:18.340970 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:18.340945    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-9sjtg_585d62ef-4d14-4ef9-8c75-c7c64fcd81dd/server/0.log"
Sep 18 20:47:18.458911 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:18.458875    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-fsngc_148d8816-4714-4a0f-9781-1ab05751e7f4/manager/0.log"
Sep 18 20:47:18.553024 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:18.552927    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-vwhkk_5e0d3150-5c7b-4d0e-87cf-754f5f56cd9d/seaweedfs/0.log"
Sep 18 20:47:19.248245 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:19.248172    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-l6zrt"
Sep 18 20:47:24.170434 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:24.170403    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hl4v9_9b2d5d99-7944-4e3c-95f7-a849c81acda1/kube-multus-additional-cni-plugins/0.log"
Sep 18 20:47:24.194487 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:24.194440    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hl4v9_9b2d5d99-7944-4e3c-95f7-a849c81acda1/egress-router-binary-copy/0.log"
Sep 18 20:47:24.216205 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:24.216153    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hl4v9_9b2d5d99-7944-4e3c-95f7-a849c81acda1/cni-plugins/0.log"
Sep 18 20:47:24.241746 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:24.241711    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hl4v9_9b2d5d99-7944-4e3c-95f7-a849c81acda1/bond-cni-plugin/0.log"
Sep 18 20:47:24.264236 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:24.264208    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hl4v9_9b2d5d99-7944-4e3c-95f7-a849c81acda1/routeoverride-cni/0.log"
Sep 18 20:47:24.287196 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:24.287168    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hl4v9_9b2d5d99-7944-4e3c-95f7-a849c81acda1/whereabouts-cni-bincopy/0.log"
Sep 18 20:47:24.306612 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:24.306581    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hl4v9_9b2d5d99-7944-4e3c-95f7-a849c81acda1/whereabouts-cni/0.log"
Sep 18 20:47:24.694291 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:24.694260    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-b6xk6_37093cb5-2840-4b9c-97b6-f66bbd717c9c/kube-multus/0.log"
Sep 18 20:47:24.788190 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:24.788152    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7kdrq_4e506cbe-bfeb-4614-b6c7-91b349785053/network-metrics-daemon/0.log"
Sep 18 20:47:24.807397 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:24.807370    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7kdrq_4e506cbe-bfeb-4614-b6c7-91b349785053/kube-rbac-proxy/0.log"
Sep 18 20:47:26.302257 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:26.302230    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-controller/0.log"
Sep 18 20:47:26.320324 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:26.320298    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/1.log"
Sep 18 20:47:26.320324 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:26.320292    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovn-acl-logging/0.log"
Sep 18 20:47:26.338886 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:26.338860    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/kube-rbac-proxy-node/0.log"
Sep 18 20:47:26.363620 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:26.363591    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 20:47:26.391147 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:26.391119    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/northd/0.log"
Sep 18 20:47:26.412342 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:26.412312    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/nbdb/0.log"
Sep 18 20:47:26.434621 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:26.434591    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/sbdb/0.log"
Sep 18 20:47:26.470393 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:26.470365    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m5vk2_c03a0417-3da2-4804-b145-74d2f4954f2c/ovnkube-controller/0.log"
Sep 18 20:47:27.524821 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:27.524792    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bm5dj_edb74921-50a4-402a-b2cc-138778d94d25/network-check-target-container/0.log"
Sep 18 20:47:28.666711 ip-10-0-138-28 kubenswrapper[2557]: I0918 20:47:28.666684    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zlggb_86c0c73b-2139-4f34-9bc6-a972fb211a9f/iptables-alerter/0.log"