Sep 16 18:43:25.911264 ip-10-0-143-139 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:43:25.911278 ip-10-0-143-139 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:43:25.911288 ip-10-0-143-139 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:43:25.911518 ip-10-0-143-139 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:43:36.005937 ip-10-0-143-139 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:43:36.005954 ip-10-0-143-139 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 67d89fe78fe74f51ab26b477adb00f95 --
Sep 16 18:45:32.689380 ip-10-0-143-139 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:45:33.107842 ip-10-0-143-139 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:33.107842 ip-10-0-143-139 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:45:33.107842 ip-10-0-143-139 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:33.107842 ip-10-0-143-139 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:45:33.107842 ip-10-0-143-139 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:33.108516 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.107858    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:45:33.112169 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112153    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:33.112169 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112168    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112172    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112175    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112178    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112186    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112189    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112192    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112195    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112198    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112201    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112204    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112207    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112210    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112212    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112214    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112217    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112219    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112222    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112226    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:33.112229 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112231    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112234    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112237    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112239    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112242    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112245    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112248    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112250    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112253    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112255    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112258    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112260    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112263    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112265    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112268    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112270    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112273    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112275    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112278    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112281    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:33.112685 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112283    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112286    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112288    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112291    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112293    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112296    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112299    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112301    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112304    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112306    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112309    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112313    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112316    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112318    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112321    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112325    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112329    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112332    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112334    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112337    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:33.113193 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112339    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112342    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112344    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112347    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112349    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112352    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112354    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112357    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112361    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112364    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112367    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112370    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112373    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112375    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112378    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112380    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112382    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112385    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112388    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:33.113747 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112390    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112395    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112397    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112400    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112402    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112405    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112408    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112411    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112413    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112416    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112418    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112421    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112423    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112426    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112428    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112431    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112434    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112437    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112439    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.112442    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:33.114295 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113140    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113149    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113152    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113155    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113158    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113162    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113165    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113168    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113170    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113173    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113175    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113179    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113182    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113184    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113187    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113192    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113196    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113200    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113203    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:33.114852 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113206    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113208    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113211    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113214    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113217    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113220    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113222    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113225    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113240    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113243    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113246    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113248    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113251    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113254    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113256    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113259    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113262    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113264    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113267    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113270    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:33.115401 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113273    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113276    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113279    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113281    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113284    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113287    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113290    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113293    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113295    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113298    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113301    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113303    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113307    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113309    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113312    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113314    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113317    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113320    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113322    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:33.115957 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113325    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113327    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113330    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113333    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113335    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113338    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113341    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113343    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113346    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113348    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113351    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113353    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113356    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113359    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113361    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113368    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113372    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113375    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113378    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113380    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:33.116495 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113383    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113386    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113388    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113391    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113393    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113396    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113399    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113401    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113404    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113406    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113410    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113412    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113416    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113420    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113423    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113427    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113430    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113433    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113436    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:33.117041 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113439    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113442    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113807    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113811    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113814    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113817    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113820    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113823    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113828    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113831    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113834    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113837    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113840    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113843    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113845    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113848    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113850    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113853    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113856    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113858    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:33.117559 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113861    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113863    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113866    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113868    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113873    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113877    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113880    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113884    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113886    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113889    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113892    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113895    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113898    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113901    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113903    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113906    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113908    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113911    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113913    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:33.118084 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113916    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113920    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113923    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113927    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113929    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113932    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113934    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113937    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113939    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113941    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113944    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113946    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113949    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113952    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113954    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113957    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113959    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113962    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113964    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113967    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:33.118674 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113969    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113972    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113974    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113977    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113979    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113982    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113984    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113987    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113989    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113992    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113994    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113997    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.113999    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114003    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114006    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114010    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114012    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114015    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114018    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:33.119225 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114020    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114023    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114025    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114028    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114030    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114033    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114036    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114040    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114043    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114045    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114048    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114051    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114053    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114056    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114058    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114061    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114063    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114066    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114068    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114070    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:33.119731 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114073    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114075    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114078    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114167    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114202    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114209    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114214    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114219    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114223    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114228    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114232    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114235    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114238    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114242    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114245    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114249    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114252    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114255    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114258    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114261    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114264    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114268    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114271    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:45:33.120246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114274    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114277    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114280    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114284    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114287    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114290    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114294    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114297    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114300    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114303    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114307    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114310    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114314    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114317    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114320    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114324    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114327    2567 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114331    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114335    2567 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114338    2567 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114341    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114344    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114347    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114351    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:45:33.120833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114354    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114357    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114360    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114363    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114366    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114369    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114371    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114374    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114377    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114380    2567 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114392    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114396    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114399    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114402    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114405    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114408    2567 flags.go:64] FLAG: --help="false"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114411    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114414    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114417    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114420    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114423    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114427    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114430    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:45:33.121433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114433    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114436    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114440    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114443    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114447    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114450    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114453    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114456    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114459    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114462    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114464    2567 flags.go:64] FLAG: --lock-file=""
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114467    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114470    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114473    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114478    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114481    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114484    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114487    2567 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114490    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114493    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114496    2567 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114499    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114503    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114506    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114510    2567 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:45:33.122047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114513    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114516    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114519    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114522    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114525    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114528    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114531    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114538    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114541    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114546    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114549    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114552    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114557    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114560    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114564    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114566    2567 flags.go:64] FLAG: --port="10250"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114569    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114572    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d1f7244feae0ed9b"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114576    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114579    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114582    2567 flags.go:64] FLAG: --register-node="true"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114584    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114588    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114591    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:45:33.122778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114594    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114597    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114600    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114603    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114606    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114609    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114612    2567 flags.go:64] FLAG: --runonce="false"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114615    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114618    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114621    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114624    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114627    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114629    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114632    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114635    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114638    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114641    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114644    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114648    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114651    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114655    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114657    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114663    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114666    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114669    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:45:33.123630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114672    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114675    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114677    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114680    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114683    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114686    2567 flags.go:64] FLAG: --v="2"
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114695    2567 flags.go:64] FLAG: --version="false"
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114699    2567 flags.go:64] FLAG: --vmodule=""
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114702    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.114706    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114812    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114815    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114818    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114821    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114823    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114826    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114829    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114831    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114834    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114837    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114841    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114844    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:33.124250 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114846    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114849    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114851    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114855    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114858    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114860    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114863    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114866    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114869    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114871    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114874    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114877    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114879    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114882    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114884    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114887    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114890    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114893    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114896    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114898    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:33.124836 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114901    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114904    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114906    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114909    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114911    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114914    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114917    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114921    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114924    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114927    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114931    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114934    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114937    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114939    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114942    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114946    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114949    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114952    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114955    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114958    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:33.125362 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114960    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114963    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114966    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114969    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114971    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114974    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114976    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114979    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114981    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114984    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114986    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114989    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114991    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114994    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.114996    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115000    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115003    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115006    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115009    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:33.125855 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115012    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115014    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115017    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115020    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115023    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115027    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115031    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115033    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115038    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115041    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115044    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115046    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115049    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115052    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115054    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115057    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115060    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115062    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115065    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115068    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:33.126421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115070    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115073    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115075    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115078    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115081    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115083    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115086    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.115089    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.115628    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.124392    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.124407    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.125293    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.125301    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125361    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:33.126912 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125368    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125373    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125376    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125380    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125383    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125385    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125388    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125391    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125394    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125396    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125399    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125402    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125405    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125408    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125410    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125413    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125416    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125418    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125421    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125423    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:33.127296 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125426    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125428    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125431    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125434    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125437    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125439    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125443    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125445    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125449    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125452    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125454    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125457    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125460    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125463    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125465    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125468    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125471    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125473    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125477    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125481    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:33.127773 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125484    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125487    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125490    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125493    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125496    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125500    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125503    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125506    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125508    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125511    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125514    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125518    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125521    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125524    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125527    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125530    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125533    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125535    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125538    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125541    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:33.128330 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125544    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125547    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125549    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125552    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125555    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125558    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125561    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125563    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125566    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125568    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125571    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125574    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125577    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125579    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125582    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125585    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125587    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125590    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125593    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:33.128810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125596    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125598    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125601    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125603    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125606    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125608    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125611    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125613    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125616    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125618    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125621    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125624    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125626    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125629    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125632    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125635    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125637    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125640    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:33.129283 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125643    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.125647    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125755    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125760    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125763    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125765    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125768    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125771    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125774    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125776    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125779    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125781    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125784    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125787    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125790    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:33.129711 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125792    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125795    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125797    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125800    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125803    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125805    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125808    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125811    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125813    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125816    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125818    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125821    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125824    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125827    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125829    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125832    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125835    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125837    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125840    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125843    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:33.130091 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125845    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125848    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125850    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125853    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125855    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125860    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125864    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125867    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125871    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125874    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125877    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125880    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125884    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125886    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125889    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125892    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125894    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125897    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125899    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:33.130612 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125902    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125905    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125907    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125910    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125912    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125916    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125918    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125921    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125923    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125926    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125929    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125932    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125934    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125937    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125939    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125942    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125945    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125947    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125950    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125953    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:33.131066 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125955    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125958    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125960    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125963    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125966    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125968    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125971    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125974    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125976    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125979    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125982    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125986    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125990    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125993    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125995    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.125998    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126000    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126003    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126006    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126008    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:33.131676 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126011    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126013    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126016    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126018    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126021    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126023    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:33.126026    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.126031    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.126150    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.130345    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.131531    2567 server.go:1019] "Starting client certificate rotation"
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.131623    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:33.132170 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.131653    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:33.134257 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.134244    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:33.136969 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.136956    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:33.147193 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.147170    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:45:33.150373 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.150355    2567 log.go:25] "Validated CRI v1 image API"
Sep 16 18:45:33.151652 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.151639    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:45:33.155381 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.155363    2567 fs.go:135] Filesystem UUIDs: map[2a08d3c1-3b46-46cd-8b5f-79a3823236d4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9ce544f3-5934-458a-ae23-a3e191345f07:/dev/nvme0n1p4]
Sep 16 18:45:33.155435 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.155381    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:45:33.160280 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.160169    2567 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:33.158874282 +0000 UTC m=+0.347320955 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3079103 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec296259ec15a3d804f3e771c818e5ba SystemUUID:ec296259-ec15-a3d8-04f3-e771c818e5ba BootID:67d89fe7-8fe7-4f51-ab26-b477adb00f95 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:3f:72:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:3f:72:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:a7:43:18:cc:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:45:33.160280 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.160269    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:45:33.160421 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.160344    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:45:33.162375 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.162359    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:33.163869 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.163844    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:45:33.164036 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.163882    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:45:33.164816 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.164804    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:45:33.164874 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.164820    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:45:33.164874 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.164832    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:33.165671 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.165659    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:33.166901 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.166889    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:45:33.171322 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.171308    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:33.171457 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.171447    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:45:33.171513 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.171483    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:45:33.171513 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.171498    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:45:33.171603 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.171520    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:45:33.171603 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.171533    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:45:33.171603 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.171546    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:45:33.171930 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.171917    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:33.171997 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.171940    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:33.176239 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.176215    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:45:33.179077 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.179064    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:45:33.179138 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.179094    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:45:33.180818 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180806    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:45:33.180859 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180823    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:45:33.180859 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180830    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:45:33.180859 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180835    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:45:33.180859 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180843    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:45:33.180859 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180850    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:45:33.180984 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180863    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:45:33.180984 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180869    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:45:33.180984 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180875    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:45:33.180984 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180881    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:45:33.180984 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180890    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:45:33.180984 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.180895    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:45:33.181644 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.181631    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:45:33.181678 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.181648    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:45:33.184575 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.184558    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:45:33.185147 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.185126    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:45:33.185187 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.185128    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:45:33.185595 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.185583    2567 server.go:1268] "Started kubelet"
Sep 16 18:45:33.185722 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.185692    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:45:33.185771 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.185690    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:45:33.185771 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.185752    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:45:33.186502 ip-10-0-143-139 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:45:33.187256 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.187239    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:45:33.187792 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.187761    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:45:33.190189 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.190172    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vcdcv"
Sep 16 18:45:33.191294 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.191277    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:45:33.192145 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.191128    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-139.ec2.internal.18d5e1782b211391  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-139.ec2.internal,UID:ip-10-0-143-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-139.ec2.internal,},FirstTimestamp:2026-09-16 18:45:33.185553297 +0000 UTC m=+0.373999970,LastTimestamp:2026-09-16 18:45:33.185553297 +0000 UTC m=+0.373999970,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-139.ec2.internal,}"
Sep 16 18:45:33.194245 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.194226    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:45:33.194245 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.194237    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:33.194381 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.194290    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:33.195218 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.194848    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:45:33.195317 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.194892    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:45:33.195317 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.195236    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:45:33.195317 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.195077    2567 factory.go:55] Registering systemd factory
Sep 16 18:45:33.195459 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.195321    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:45:33.195459 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.195387    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:45:33.195459 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.195397    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:45:33.195609 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.195588    2567 factory.go:153] Registering CRI-O factory
Sep 16 18:45:33.195609 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.195608    2567 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:45:33.195770 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.195671    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:45:33.195770 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.195689    2567 factory.go:103] Registering Raw factory
Sep 16 18:45:33.195770 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.195704    2567 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:45:33.196030 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.196016    2567 manager.go:349] Starting recovery of all containers
Sep 16 18:45:33.197854 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.197831    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:45:33.197957 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.197923    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:45:33.201797 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.201771    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vcdcv"
Sep 16 18:45:33.207219 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.207102    2567 manager.go:354] Recovery completed
Sep 16 18:45:33.212588 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.212573    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:33.214489 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.214474    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:33.214561 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.214501    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:33.214561 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.214511    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:33.215015 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.214997    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:45:33.215015 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.215007    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:33.215015 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.215012    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:45:33.215233 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.215030    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:33.215655 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.215643    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:33.215703 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.215666    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:33.215703 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.215679    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:33.216720 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.216649    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-139.ec2.internal.18d5e1782cda9a45  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-139.ec2.internal,UID:ip-10-0-143-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-139.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-139.ec2.internal,},FirstTimestamp:2026-09-16 18:45:33.214489157 +0000 UTC m=+0.402935830,LastTimestamp:2026-09-16 18:45:33.214489157 +0000 UTC m=+0.402935830,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-139.ec2.internal,}"
Sep 16 18:45:33.216968 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.216956    2567 policy_none.go:49] "None policy: Start"
Sep 16 18:45:33.217022 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.216974    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:45:33.217022 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.216987    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:45:33.218910 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.218897    2567 policy_none.go:47] "Start"
Sep 16 18:45:33.251758 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.251743    2567 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:45:33.273187 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.251777    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:45:33.273187 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.251791    2567 server.go:87] "Starting device plugin registration server"
Sep 16 18:45:33.273187 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.252095    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:45:33.273187 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.252122    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:45:33.273187 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.252238    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:45:33.273187 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.252313    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:45:33.273187 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.252321    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:45:33.273187 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.252705    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:45:33.273187 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.252759    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:33.326577 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.326547    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:45:33.327728 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.327714    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:45:33.327788 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.327759    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:45:33.327788 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.327778    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:45:33.327868 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.327813    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:45:33.329747 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.329729    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:33.352497 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.352482    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:33.354944 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.354430    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:33.354944 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.354459    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:33.354944 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.354473    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:33.354944 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.354501    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.362851 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.362772    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.362851 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.362792    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-139.ec2.internal\": node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:33.373106 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.373090    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:33.428138 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.428095    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"]
Sep 16 18:45:33.428230 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.428174    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:33.428960 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.428945    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:33.429029 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.428970    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:33.429029 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.428980    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:33.430986 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.430974    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:33.431102 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.431089    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.431177 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.431128    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:33.431653 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.431638    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:33.431722 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.431652    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:33.431722 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.431664    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:33.431722 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.431673    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:33.431722 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.431684    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:33.431722 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.431676    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:33.433686 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.433672    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.433747 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.433695    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:33.435220 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.435205    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:33.435307 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.435234    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:33.435307 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.435248    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:33.450639 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.450619    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-139.ec2.internal\" not found" node="ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.455022 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.455002    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-139.ec2.internal\" not found" node="ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.474030 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.474013    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:33.574673 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.574650    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:33.596961 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.596940    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd97510a049e6f9ef06e726eab9ef77-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ccd97510a049e6f9ef06e726eab9ef77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.597043 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.596970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd53a7a796c4569b9de0a17cd4101459-config\") pod \"kube-apiserver-proxy-ip-10-0-143-139.ec2.internal\" (UID: \"cd53a7a796c4569b9de0a17cd4101459\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.597043 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.597000    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ccd97510a049e6f9ef06e726eab9ef77-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ccd97510a049e6f9ef06e726eab9ef77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.675005 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.674981    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:33.697348 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.697322    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd53a7a796c4569b9de0a17cd4101459-config\") pod \"kube-apiserver-proxy-ip-10-0-143-139.ec2.internal\" (UID: \"cd53a7a796c4569b9de0a17cd4101459\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.697348 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.697352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ccd97510a049e6f9ef06e726eab9ef77-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ccd97510a049e6f9ef06e726eab9ef77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.697478 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.697376    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd97510a049e6f9ef06e726eab9ef77-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ccd97510a049e6f9ef06e726eab9ef77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.697478 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.697408    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd97510a049e6f9ef06e726eab9ef77-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ccd97510a049e6f9ef06e726eab9ef77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.697478 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.697424    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd53a7a796c4569b9de0a17cd4101459-config\") pod \"kube-apiserver-proxy-ip-10-0-143-139.ec2.internal\" (UID: \"cd53a7a796c4569b9de0a17cd4101459\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.697478 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.697429    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ccd97510a049e6f9ef06e726eab9ef77-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ccd97510a049e6f9ef06e726eab9ef77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.757152 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.757111    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.759384 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:33.759363    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:33.775828 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.775808    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:33.880150 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.880095    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:33.980481 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:33.980405    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:34.080567 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:34.080536    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:34.131477 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.131450    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:45:34.131863 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.131583    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:34.181665 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:34.181645    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:34.213051 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.213020    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:33 +0000 UTC" deadline="2028-06-20 04:48:52.782259084 +0000 UTC"
Sep 16 18:45:34.213051 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.213048    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15418h3m18.569213769s"
Sep 16 18:45:34.282651 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:34.282586    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:34.382965 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:34.382936    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:34.407237 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.407207    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:34.477733 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:34.477693    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd53a7a796c4569b9de0a17cd4101459.slice/crio-2a4f560b22e5d980675aabf3e90cd88890324ef13f2f7831e00d2e43f91f211f WatchSource:0}: Error finding container 2a4f560b22e5d980675aabf3e90cd88890324ef13f2f7831e00d2e43f91f211f: Status 404 returned error can't find the container with id 2a4f560b22e5d980675aabf3e90cd88890324ef13f2f7831e00d2e43f91f211f
Sep 16 18:45:34.478047 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:34.478025    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd97510a049e6f9ef06e726eab9ef77.slice/crio-28e06c97a8f231325c539706f0ca6b955632cee42a8c6667b1cce2972f09511d WatchSource:0}: Error finding container 28e06c97a8f231325c539706f0ca6b955632cee42a8c6667b1cce2972f09511d: Status 404 returned error can't find the container with id 28e06c97a8f231325c539706f0ca6b955632cee42a8c6667b1cce2972f09511d
Sep 16 18:45:34.483932 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:34.483912    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:34.484293 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.484273    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:34.586240 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:34.586161    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Sep 16 18:45:34.659910 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.659882    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:34.694861 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.694834    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:34.708160 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.708143    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:34.709922 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.709907    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Sep 16 18:45:34.721924 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.721904    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:34.727028 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:34.727009    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:35.021178 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.021145    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:35.176284 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.176255    2567 apiserver.go:52] "Watching apiserver"
Sep 16 18:45:35.188339 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.188304    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:45:35.188820 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.188790    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-fbnt4","kube-system/global-pull-secret-syncer-8g5df","kube-system/konnectivity-agent-wqzk6","openshift-cluster-node-tuning-operator/tuned-qxvlt","openshift-dns/node-resolver-bhsmz","openshift-network-diagnostics/network-check-target-g8wrg","openshift-ovn-kubernetes/ovnkube-node-69rtg","kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs","openshift-image-registry/node-ca-zbkkp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal","openshift-multus/multus-additional-cni-plugins-vfxbv","openshift-multus/multus-p2z8b","openshift-multus/network-metrics-daemon-fsv9w"]
Sep 16 18:45:35.191617 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.191590    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.194049 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.194027    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.195179 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.195156    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:45:35.195450 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.195432    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:35.195948 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.195931    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:45:35.196519 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.196500    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:45:35.196618 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.196518    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mv9lz\""
Sep 16 18:45:35.196747 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.196730    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:45:35.196818 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.196763    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.196889 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.196836    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.197398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.197379    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:45:35.197483 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.197408    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wvp8q\""
Sep 16 18:45:35.197483 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.197445    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:35.197590 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.197526    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:45:35.197845 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.197824    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.199870 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.199843    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.201367 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.201344    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:45:35.201453 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.201368    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:45:35.201453 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.201442    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.201619 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.201601    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rhzj9\""
Sep 16 18:45:35.202908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.202886    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fnpc8\""
Sep 16 18:45:35.203614 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.203595    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.204624 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.204605    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.204927 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.204907    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.205879 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.205857    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-systemd-units\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.205965 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.205893    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-node-log\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.205965 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.205901    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:35.205965 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.205922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.205965 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.205952    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.206111 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.205964    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:35.206111 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.205978    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-etc-kubernetes\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.206111 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206005    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbcd\" (UniqueName: \"kubernetes.io/projected/8856e3f1-9640-4dc7-b3fd-6936fc04f239-kube-api-access-kbbcd\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.206111 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206025    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7808bd97-d8b5-43a5-a11f-37dbbf196633-agent-certs\") pod \"konnectivity-agent-wqzk6\" (UID: \"7808bd97-d8b5-43a5-a11f-37dbbf196633\") " pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:35.206111 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206043    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7808bd97-d8b5-43a5-a11f-37dbbf196633-konnectivity-ca\") pod \"konnectivity-agent-wqzk6\" (UID: \"7808bd97-d8b5-43a5-a11f-37dbbf196633\") " pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:35.206111 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206084    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-etc-openvswitch\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.206111 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206101    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8856e3f1-9640-4dc7-b3fd-6936fc04f239-cni-binary-copy\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206119    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-var-lib-cni-multus\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206134    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-run-systemd\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206148    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-var-lib-openvswitch\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206164    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-ovn-node-metrics-cert\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206179    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-var-lib-cni-bin\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206222    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-run-netns\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206262    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-ovnkube-config\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206296    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-daemon-config\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206323    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-run-multus-certs\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206371    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-run-openvswitch\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206408    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-log-socket\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.206439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206441    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-run-k8s-cni-cncf-io\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206466    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-conf-dir\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206502    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-kubelet\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206531    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-env-overrides\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206555    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-system-cni-dir\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206581    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-cnibin\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206606    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-cni-bin\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206629    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-cni-netd\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206655    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-ovnkube-script-lib\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206679    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbvw\" (UniqueName: \"kubernetes.io/projected/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-kube-api-access-ztbvw\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206706    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-cni-dir\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206722    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206731    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-socket-dir-parent\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206734    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mgw2g\""
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206709    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206792    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-run-netns\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206826    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-slash\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206864    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-run-ovn\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206890    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-var-lib-kubelet\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.207098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.206916    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-hostroot\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.208458 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.208438    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.210996 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.210941    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:35.211125 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.211033    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:35.211526 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.211504    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.211614 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.211573    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vfb2p\""
Sep 16 18:45:35.211750 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.211735    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:45:35.212034 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.212015    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.213541 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.213522    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.215415 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.215386    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:35.216447 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.216006    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.217347 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.217322    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:45:35.217453 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.217435    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:45:35.217617 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.217591    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l5pkz\""
Sep 16 18:45:35.218761 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.218738    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.220754 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.220425    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.220754 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.220433    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:45:35.220754 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.220547    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cgnpg\""
Sep 16 18:45:35.222143 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.222109    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:35.222264 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.222203    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:35.224717 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.224695    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.228750 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.228625    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.229458 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.229429    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:45:35.229536 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.229472    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.229765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.229747    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mgldg\""
Sep 16 18:45:35.253259 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.253236    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wrkb5"
Sep 16 18:45:35.269920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.269892    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wrkb5"
Sep 16 18:45:35.297130 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.297057    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:45:35.308016 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.307992    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e69d4209-41fd-4daf-8d4b-e9fd7f377777-host\") pod \"node-ca-zbkkp\" (UID: \"e69d4209-41fd-4daf-8d4b-e9fd7f377777\") " pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.308155 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308027    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-run-openvswitch\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.308155 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-log-socket\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.308155 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308076    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-run-k8s-cni-cncf-io\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.308155 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-registration-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.308155 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-system-cni-dir\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308160    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-run-openvswitch\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308171    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-run\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308196    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-host\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308230    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-cnibin\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308241    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-log-socket\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308255    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308285    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-cnibin\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308292    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-run-k8s-cni-cncf-io\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308310    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbvw\" (UniqueName: \"kubernetes.io/projected/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-kube-api-access-ztbvw\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308338    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-socket-dir-parent\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308355    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-run-netns\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.308422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308380    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-cnibin\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308430    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-socket-dir-parent\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308439    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-run-netns\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308440    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-var-lib-kubelet\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-system-cni-dir\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-var-lib-kubelet\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308526    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-socket-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308533    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-var-lib-kubelet\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308553    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzmb5\" (UniqueName: \"kubernetes.io/projected/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-kube-api-access-qzmb5\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308573    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea444e57-cd1a-4620-a3aa-df7e8348c481-tmp-dir\") pod \"node-resolver-bhsmz\" (UID: \"ea444e57-cd1a-4620-a3aa-df7e8348c481\") " pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308589    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-node-log\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308608    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308632    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-etc-kubernetes\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbcd\" (UniqueName: \"kubernetes.io/projected/8856e3f1-9640-4dc7-b3fd-6936fc04f239-kube-api-access-kbbcd\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308676    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308681    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-modprobe-d\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308685    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-etc-kubernetes\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.309031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308676    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-node-log\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308707    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-sysconfig\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308755    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-lib-modules\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308789    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a858b275-e882-46fc-b2c4-164767aaea80-tmp\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308829    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-etc-openvswitch\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308883    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-etc-openvswitch\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308892    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-sys-fs\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nz5\" (UniqueName: \"kubernetes.io/projected/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-kube-api-access-h8nz5\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308949    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.308973    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvp6m\" (UniqueName: \"kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m\") pod \"network-check-target-g8wrg\" (UID: \"a17239c3-07cb-400b-ac0c-36941de91c01\") " pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309002    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4zr\" (UniqueName: \"kubernetes.io/projected/e7fa29a5-39f2-485f-b1fe-268334ec4935-kube-api-access-rq4zr\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309022    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7b9h\" (UniqueName: \"kubernetes.io/projected/e69d4209-41fd-4daf-8d4b-e9fd7f377777-kube-api-access-r7b9h\") pod \"node-ca-zbkkp\" (UID: \"e69d4209-41fd-4daf-8d4b-e9fd7f377777\") " pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-run-systemd\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309079    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-var-lib-openvswitch\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-ovn-node-metrics-cert\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.309908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309156    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-run-systemd\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309169    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-var-lib-cni-bin\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309196    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-var-lib-openvswitch\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309230    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc7mt\" (UniqueName: \"kubernetes.io/projected/a858b275-e882-46fc-b2c4-164767aaea80-kube-api-access-wc7mt\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309258    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2104af8-40ea-48de-8403-6c112903ca6d-dbus\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309291    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-var-lib-cni-bin\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-ovnkube-config\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309356    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-daemon-config\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9884\" (UniqueName: \"kubernetes.io/projected/ea444e57-cd1a-4620-a3aa-df7e8348c481-kube-api-access-d9884\") pod \"node-resolver-bhsmz\" (UID: \"ea444e57-cd1a-4620-a3aa-df7e8348c481\") " pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309515    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-conf-dir\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309565    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-conf-dir\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309594    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-sys\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309617    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ea444e57-cd1a-4620-a3aa-df7e8348c481-hosts-file\") pod \"node-resolver-bhsmz\" (UID: \"ea444e57-cd1a-4620-a3aa-df7e8348c481\") " pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-kubelet\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309715    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-env-overrides\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309767    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-kubelet\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309795    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a858b275-e882-46fc-b2c4-164767aaea80-etc-tuned\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309821    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2104af8-40ea-48de-8403-6c112903ca6d-kubelet-config\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309849    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-cni-bin\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-cni-netd\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309931    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-ovnkube-script-lib\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.309958    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-cni-dir\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310001    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-hostroot\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310006    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-slash\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-run-ovn\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310138    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-sysctl-d\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310221    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-ovnkube-config\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310233    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-device-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310247    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-run-ovn\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310262    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-system-cni-dir\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310293    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-cni-bin\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.311564 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310309    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e69d4209-41fd-4daf-8d4b-e9fd7f377777-serviceca\") pod \"node-ca-zbkkp\" (UID: \"e69d4209-41fd-4daf-8d4b-e9fd7f377777\") " pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310330    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-cni-dir\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310340    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-cni-netd\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310340    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4k58\" (UniqueName: \"kubernetes.io/projected/60a41c27-f58c-4070-b3bb-1172234dba44-kube-api-access-x4k58\") pod \"iptables-alerter-fbnt4\" (UID: \"60a41c27-f58c-4070-b3bb-1172234dba44\") " pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310367    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-slash\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310383    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-hostroot\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310389    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-systemd-units\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310421    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60a41c27-f58c-4070-b3bb-1172234dba44-host-slash\") pod \"iptables-alerter-fbnt4\" (UID: \"60a41c27-f58c-4070-b3bb-1172234dba44\") " pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310499    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7808bd97-d8b5-43a5-a11f-37dbbf196633-agent-certs\") pod \"konnectivity-agent-wqzk6\" (UID: \"7808bd97-d8b5-43a5-a11f-37dbbf196633\") " pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310527    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7808bd97-d8b5-43a5-a11f-37dbbf196633-konnectivity-ca\") pod \"konnectivity-agent-wqzk6\" (UID: \"7808bd97-d8b5-43a5-a11f-37dbbf196633\") " pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310552    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8856e3f1-9640-4dc7-b3fd-6936fc04f239-cni-binary-copy\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310580    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-var-lib-cni-multus\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310606    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-systemd\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310633    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310661    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/60a41c27-f58c-4070-b3bb-1172234dba44-iptables-alerter-script\") pod \"iptables-alerter-fbnt4\" (UID: \"60a41c27-f58c-4070-b3bb-1172234dba44\") " pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310687    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:35.312438 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310766    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-run-netns\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310796    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-run-multus-certs\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310805    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-env-overrides\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-kubernetes\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310854    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-sysctl-conf\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310863    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-ovnkube-script-lib\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310881    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.310941    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8856e3f1-9640-4dc7-b3fd-6936fc04f239-multus-daemon-config\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.311009    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-var-lib-cni-multus\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.311017    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8856e3f1-9640-4dc7-b3fd-6936fc04f239-host-run-multus-certs\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.311048    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-run-netns\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.311111    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.311121    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-systemd-units\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.311287    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8856e3f1-9640-4dc7-b3fd-6936fc04f239-cni-binary-copy\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.313277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.311296    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7808bd97-d8b5-43a5-a11f-37dbbf196633-konnectivity-ca\") pod \"konnectivity-agent-wqzk6\" (UID: \"7808bd97-d8b5-43a5-a11f-37dbbf196633\") " pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:35.314609 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.314577    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-ovn-node-metrics-cert\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.314770 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.314723    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7808bd97-d8b5-43a5-a11f-37dbbf196633-agent-certs\") pod \"konnectivity-agent-wqzk6\" (UID: \"7808bd97-d8b5-43a5-a11f-37dbbf196633\") " pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:35.329702 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.329610    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbvw\" (UniqueName: \"kubernetes.io/projected/6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c-kube-api-access-ztbvw\") pod \"ovnkube-node-69rtg\" (UID: \"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.332831 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.332805    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbcd\" (UniqueName: \"kubernetes.io/projected/8856e3f1-9640-4dc7-b3fd-6936fc04f239-kube-api-access-kbbcd\") pod \"multus-p2z8b\" (UID: \"8856e3f1-9640-4dc7-b3fd-6936fc04f239\") " pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.333537 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.333488    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" event={"ID":"cd53a7a796c4569b9de0a17cd4101459","Type":"ContainerStarted","Data":"2a4f560b22e5d980675aabf3e90cd88890324ef13f2f7831e00d2e43f91f211f"}
Sep 16 18:45:35.335038 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.335008    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" event={"ID":"ccd97510a049e6f9ef06e726eab9ef77","Type":"ContainerStarted","Data":"28e06c97a8f231325c539706f0ca6b955632cee42a8c6667b1cce2972f09511d"}
Sep 16 18:45:35.411227 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411178    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea444e57-cd1a-4620-a3aa-df7e8348c481-tmp-dir\") pod \"node-resolver-bhsmz\" (UID: \"ea444e57-cd1a-4620-a3aa-df7e8348c481\") " pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.411227 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411229    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-modprobe-d\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.411461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411255    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-sysconfig\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.411461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411279    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-lib-modules\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.411461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a858b275-e882-46fc-b2c4-164767aaea80-tmp\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.411461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411327    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.411461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-sys-fs\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.411461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411379    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nz5\" (UniqueName: \"kubernetes.io/projected/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-kube-api-access-h8nz5\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.411461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411405    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:35.411461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411431    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp6m\" (UniqueName: \"kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m\") pod \"network-check-target-g8wrg\" (UID: \"a17239c3-07cb-400b-ac0c-36941de91c01\") " pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:35.411461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411456    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4zr\" (UniqueName: \"kubernetes.io/projected/e7fa29a5-39f2-485f-b1fe-268334ec4935-kube-api-access-rq4zr\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411482    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b9h\" (UniqueName: \"kubernetes.io/projected/e69d4209-41fd-4daf-8d4b-e9fd7f377777-kube-api-access-r7b9h\") pod \"node-ca-zbkkp\" (UID: \"e69d4209-41fd-4daf-8d4b-e9fd7f377777\") " pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc7mt\" (UniqueName: \"kubernetes.io/projected/a858b275-e882-46fc-b2c4-164767aaea80-kube-api-access-wc7mt\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2104af8-40ea-48de-8403-6c112903ca6d-dbus\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411537    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9884\" (UniqueName: \"kubernetes.io/projected/ea444e57-cd1a-4620-a3aa-df7e8348c481-kube-api-access-d9884\") pod \"node-resolver-bhsmz\" (UID: \"ea444e57-cd1a-4620-a3aa-df7e8348c481\") " pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411552    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-sys\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411566    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ea444e57-cd1a-4620-a3aa-df7e8348c481-hosts-file\") pod \"node-resolver-bhsmz\" (UID: \"ea444e57-cd1a-4620-a3aa-df7e8348c481\") " pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a858b275-e882-46fc-b2c4-164767aaea80-etc-tuned\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2104af8-40ea-48de-8403-6c112903ca6d-kubelet-config\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411649    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-sysctl-d\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411652    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea444e57-cd1a-4620-a3aa-df7e8348c481-tmp-dir\") pod \"node-resolver-bhsmz\" (UID: \"ea444e57-cd1a-4620-a3aa-df7e8348c481\") " pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411667    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-device-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-system-cni-dir\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e69d4209-41fd-4daf-8d4b-e9fd7f377777-serviceca\") pod \"node-ca-zbkkp\" (UID: \"e69d4209-41fd-4daf-8d4b-e9fd7f377777\") " pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4k58\" (UniqueName: \"kubernetes.io/projected/60a41c27-f58c-4070-b3bb-1172234dba44-kube-api-access-x4k58\") pod \"iptables-alerter-fbnt4\" (UID: \"60a41c27-f58c-4070-b3bb-1172234dba44\") " pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.411918 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411776    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60a41c27-f58c-4070-b3bb-1172234dba44-host-slash\") pod \"iptables-alerter-fbnt4\" (UID: \"60a41c27-f58c-4070-b3bb-1172234dba44\") " pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411806    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-systemd\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411826    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411850    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/60a41c27-f58c-4070-b3bb-1172234dba44-iptables-alerter-script\") pod \"iptables-alerter-fbnt4\" (UID: \"60a41c27-f58c-4070-b3bb-1172234dba44\") " pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411919    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-sysconfig\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411961    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411966    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.411995    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-kubernetes\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412001    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-modprobe-d\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412023    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-sysctl-conf\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412050    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412075    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e69d4209-41fd-4daf-8d4b-e9fd7f377777-host\") pod \"node-ca-zbkkp\" (UID: \"e69d4209-41fd-4daf-8d4b-e9fd7f377777\") " pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412088    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-lib-modules\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412106    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-registration-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412145    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-sys-fs\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412155    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-run\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412164    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-kubernetes\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.412787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412180    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-host\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-cnibin\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412220    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e69d4209-41fd-4daf-8d4b-e9fd7f377777-host\") pod \"node-ca-zbkkp\" (UID: \"e69d4209-41fd-4daf-8d4b-e9fd7f377777\") " pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412237    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412258    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-sysctl-conf\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.412093    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412276    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-var-lib-kubelet\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-socket-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412313    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-host\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzmb5\" (UniqueName: \"kubernetes.io/projected/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-kube-api-access-qzmb5\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412377    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-run\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.412395    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs podName:e7fa29a5-39f2-485f-b1fe-268334ec4935 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.912370915 +0000 UTC m=+3.100817605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs") pod "network-metrics-daemon-fsv9w" (UID: "e7fa29a5-39f2-485f-b1fe-268334ec4935") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412409    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-sys\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412427    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412440    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-registration-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.412094    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412465    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ea444e57-cd1a-4620-a3aa-df7e8348c481-hosts-file\") pod \"node-resolver-bhsmz\" (UID: \"ea444e57-cd1a-4620-a3aa-df7e8348c481\") " pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.413654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-device-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.412500    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret podName:a2104af8-40ea-48de-8403-6c112903ca6d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.912488036 +0000 UTC m=+3.100934729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret") pod "global-pull-secret-syncer-8g5df" (UID: "a2104af8-40ea-48de-8403-6c112903ca6d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412520    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-system-cni-dir\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-var-lib-kubelet\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412596    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60a41c27-f58c-4070-b3bb-1172234dba44-host-slash\") pod \"iptables-alerter-fbnt4\" (UID: \"60a41c27-f58c-4070-b3bb-1172234dba44\") " pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412623    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/60a41c27-f58c-4070-b3bb-1172234dba44-iptables-alerter-script\") pod \"iptables-alerter-fbnt4\" (UID: \"60a41c27-f58c-4070-b3bb-1172234dba44\") " pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412648    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-systemd\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412675    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-socket-dir\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412694    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2104af8-40ea-48de-8403-6c112903ca6d-dbus\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.412744    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2104af8-40ea-48de-8403-6c112903ca6d-kubelet-config\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.413104    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.413130    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e69d4209-41fd-4daf-8d4b-e9fd7f377777-serviceca\") pod \"node-ca-zbkkp\" (UID: \"e69d4209-41fd-4daf-8d4b-e9fd7f377777\") " pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.413159    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-cnibin\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.413264    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.413292    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a858b275-e882-46fc-b2c4-164767aaea80-etc-sysctl-d\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.413309    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.414303 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.413693    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.415122 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.415002    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a858b275-e882-46fc-b2c4-164767aaea80-etc-tuned\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.415175 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.415111    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a858b275-e882-46fc-b2c4-164767aaea80-tmp\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.426810 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.426784    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4zr\" (UniqueName: \"kubernetes.io/projected/e7fa29a5-39f2-485f-b1fe-268334ec4935-kube-api-access-rq4zr\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:35.427267 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.427223    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:35.427267 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.427249    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:35.427267 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.427263    2567 projected.go:196] Error preparing data for projected volume kube-api-access-jvp6m for pod openshift-network-diagnostics/network-check-target-g8wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:35.427484 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.427324    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m podName:a17239c3-07cb-400b-ac0c-36941de91c01 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.927305871 +0000 UTC m=+3.115752539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jvp6m" (UniqueName: "kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m") pod "network-check-target-g8wrg" (UID: "a17239c3-07cb-400b-ac0c-36941de91c01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:35.428593 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.428556    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nz5\" (UniqueName: \"kubernetes.io/projected/9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298-kube-api-access-h8nz5\") pod \"multus-additional-cni-plugins-vfxbv\" (UID: \"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298\") " pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.428868 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.428845    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4k58\" (UniqueName: \"kubernetes.io/projected/60a41c27-f58c-4070-b3bb-1172234dba44-kube-api-access-x4k58\") pod \"iptables-alerter-fbnt4\" (UID: \"60a41c27-f58c-4070-b3bb-1172234dba44\") " pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.431755 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.431731    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7b9h\" (UniqueName: \"kubernetes.io/projected/e69d4209-41fd-4daf-8d4b-e9fd7f377777-kube-api-access-r7b9h\") pod \"node-ca-zbkkp\" (UID: \"e69d4209-41fd-4daf-8d4b-e9fd7f377777\") " pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.432237 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.432216    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc7mt\" (UniqueName: \"kubernetes.io/projected/a858b275-e882-46fc-b2c4-164767aaea80-kube-api-access-wc7mt\") pod \"tuned-qxvlt\" (UID: \"a858b275-e882-46fc-b2c4-164767aaea80\") " pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.432920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.432900    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9884\" (UniqueName: \"kubernetes.io/projected/ea444e57-cd1a-4620-a3aa-df7e8348c481-kube-api-access-d9884\") pod \"node-resolver-bhsmz\" (UID: \"ea444e57-cd1a-4620-a3aa-df7e8348c481\") " pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.433821 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.433799    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzmb5\" (UniqueName: \"kubernetes.io/projected/af918883-ce09-4d8b-ac4b-d7fb7b190dc4-kube-api-access-qzmb5\") pod \"aws-ebs-csi-driver-node-qfsvs\" (UID: \"af918883-ce09-4d8b-ac4b-d7fb7b190dc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.504964 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.504929    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:45:35.513206 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.513179    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2z8b"
Sep 16 18:45:35.525682 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.525655    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:35.532916 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.532893    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qxvlt"
Sep 16 18:45:35.541087 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.541064    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bhsmz"
Sep 16 18:45:35.549474 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.549417    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs"
Sep 16 18:45:35.557766 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.557742    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vfxbv"
Sep 16 18:45:35.567437 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.567418    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zbkkp"
Sep 16 18:45:35.574421 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.574400    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fbnt4"
Sep 16 18:45:35.916208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.916161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:35.916415 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.916339    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:35.916415 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:35.916343    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:35.916532 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.916416    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:35.916532 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.916422    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret podName:a2104af8-40ea-48de-8403-6c112903ca6d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:36.916400513 +0000 UTC m=+4.104847181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret") pod "global-pull-secret-syncer-8g5df" (UID: "a2104af8-40ea-48de-8403-6c112903ca6d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:35.916532 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:35.916467    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs podName:e7fa29a5-39f2-485f-b1fe-268334ec4935 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:36.916450681 +0000 UTC m=+4.104897370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs") pod "network-metrics-daemon-fsv9w" (UID: "e7fa29a5-39f2-485f-b1fe-268334ec4935") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:36.017407 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:36.017375    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp6m\" (UniqueName: \"kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m\") pod \"network-check-target-g8wrg\" (UID: \"a17239c3-07cb-400b-ac0c-36941de91c01\") " pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:36.017608 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:36.017512    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:36.017608 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:36.017525    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:36.017608 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:36.017534    2567 projected.go:196] Error preparing data for projected volume kube-api-access-jvp6m for pod openshift-network-diagnostics/network-check-target-g8wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:36.017608 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:36.017579    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m podName:a17239c3-07cb-400b-ac0c-36941de91c01 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:37.017565793 +0000 UTC m=+4.206012455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvp6m" (UniqueName: "kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m") pod "network-check-target-g8wrg" (UID: "a17239c3-07cb-400b-ac0c-36941de91c01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:36.273937 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:36.273819    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:35 +0000 UTC" deadline="2028-04-25 12:45:27.31059067 +0000 UTC"
Sep 16 18:45:36.273937 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:36.273883    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14081h59m51.036713709s"
Sep 16 18:45:36.465043 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:36.465004    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6613b2_b8d0_48a1_bfd2_3eaeca2ae298.slice/crio-eca67b726eea5e6df0bfab949a1ff4d48a987f30bfe5a18d83641a8252643e16 WatchSource:0}: Error finding container eca67b726eea5e6df0bfab949a1ff4d48a987f30bfe5a18d83641a8252643e16: Status 404 returned error can't find the container with id eca67b726eea5e6df0bfab949a1ff4d48a987f30bfe5a18d83641a8252643e16
Sep 16 18:45:36.465590 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:36.465566    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda858b275_e882_46fc_b2c4_164767aaea80.slice/crio-bfa9831e19910e1762c591ec5e967631bfe0e19ce1d3944396d34ac43788b14a WatchSource:0}: Error finding container bfa9831e19910e1762c591ec5e967631bfe0e19ce1d3944396d34ac43788b14a: Status 404 returned error can't find the container with id bfa9831e19910e1762c591ec5e967631bfe0e19ce1d3944396d34ac43788b14a
Sep 16 18:45:36.468478 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:36.468442    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea444e57_cd1a_4620_a3aa_df7e8348c481.slice/crio-1e76e0db4fb7aa65a69db3172e88fa032b69a4a830fb109388a5242061c48729 WatchSource:0}: Error finding container 1e76e0db4fb7aa65a69db3172e88fa032b69a4a830fb109388a5242061c48729: Status 404 returned error can't find the container with id 1e76e0db4fb7aa65a69db3172e88fa032b69a4a830fb109388a5242061c48729
Sep 16 18:45:36.470485 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:36.470461    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8856e3f1_9640_4dc7_b3fd_6936fc04f239.slice/crio-1c29c395f5b261da25f3d06c1fafcf28cf964f023cf6d69e2934628dde57f7bf WatchSource:0}: Error finding container 1c29c395f5b261da25f3d06c1fafcf28cf964f023cf6d69e2934628dde57f7bf: Status 404 returned error can't find the container with id 1c29c395f5b261da25f3d06c1fafcf28cf964f023cf6d69e2934628dde57f7bf
Sep 16 18:45:36.472409 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:36.472391    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf918883_ce09_4d8b_ac4b_d7fb7b190dc4.slice/crio-0e6ebdc016040a91cd256f4bc5e464b60b79061f248b77a61d89a0816855f00a WatchSource:0}: Error finding container 0e6ebdc016040a91cd256f4bc5e464b60b79061f248b77a61d89a0816855f00a: Status 404 returned error can't find the container with id 0e6ebdc016040a91cd256f4bc5e464b60b79061f248b77a61d89a0816855f00a
Sep 16 18:45:36.473267 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:36.473238    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7808bd97_d8b5_43a5_a11f_37dbbf196633.slice/crio-20d92eab4b4145c6f752e38418eeec9a1b0eb8f6f6d12278b6b1646595913db7 WatchSource:0}: Error finding container 20d92eab4b4145c6f752e38418eeec9a1b0eb8f6f6d12278b6b1646595913db7: Status 404 returned error can't find the container with id 20d92eab4b4145c6f752e38418eeec9a1b0eb8f6f6d12278b6b1646595913db7
Sep 16 18:45:36.474140 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:36.474057    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a41c27_f58c_4070_b3bb_1172234dba44.slice/crio-93a6f4b5c973bea7d909a897cb4c722b0f540aabd8031af7d847aaa34a486a85 WatchSource:0}: Error finding container 93a6f4b5c973bea7d909a897cb4c722b0f540aabd8031af7d847aaa34a486a85: Status 404 returned error can't find the container with id 93a6f4b5c973bea7d909a897cb4c722b0f540aabd8031af7d847aaa34a486a85
Sep 16 18:45:36.475044 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:36.475020    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef3304e_ccf6_4f23_b0f6_6b9aab4cdb2c.slice/crio-fd0cc1804165fbf85f5807487e1695c0be04529d8acfa076305dd657ae9182b1 WatchSource:0}: Error finding container fd0cc1804165fbf85f5807487e1695c0be04529d8acfa076305dd657ae9182b1: Status 404 returned error can't find the container with id fd0cc1804165fbf85f5807487e1695c0be04529d8acfa076305dd657ae9182b1
Sep 16 18:45:36.478080 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:45:36.477051    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69d4209_41fd_4daf_8d4b_e9fd7f377777.slice/crio-1d2b3e5b2aa4d7099c974a90bc5c05d0c4f190d10d3ab4b8bad2ee09f5432129 WatchSource:0}: Error finding container 1d2b3e5b2aa4d7099c974a90bc5c05d0c4f190d10d3ab4b8bad2ee09f5432129: Status 404 returned error can't find the container with id 1d2b3e5b2aa4d7099c974a90bc5c05d0c4f190d10d3ab4b8bad2ee09f5432129
Sep 16 18:45:36.923838 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:36.923797    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:36.924033 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:36.923870    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:36.924033 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:36.923955    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:36.924033 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:36.923990    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:36.924238 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:36.924035    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs podName:e7fa29a5-39f2-485f-b1fe-268334ec4935 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:38.924014037 +0000 UTC m=+6.112460713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs") pod "network-metrics-daemon-fsv9w" (UID: "e7fa29a5-39f2-485f-b1fe-268334ec4935") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:36.924238 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:36.924058    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret podName:a2104af8-40ea-48de-8403-6c112903ca6d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:38.924047247 +0000 UTC m=+6.112493913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret") pod "global-pull-secret-syncer-8g5df" (UID: "a2104af8-40ea-48de-8403-6c112903ca6d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:37.024921 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.024879    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp6m\" (UniqueName: \"kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m\") pod \"network-check-target-g8wrg\" (UID: \"a17239c3-07cb-400b-ac0c-36941de91c01\") " pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:37.025083 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:37.025061    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:37.025123 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:37.025097    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:37.025123 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:37.025111    2567 projected.go:196] Error preparing data for projected volume kube-api-access-jvp6m for pod openshift-network-diagnostics/network-check-target-g8wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:37.025237 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:37.025191    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m podName:a17239c3-07cb-400b-ac0c-36941de91c01 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.025170865 +0000 UTC m=+6.213617544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvp6m" (UniqueName: "kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m") pod "network-check-target-g8wrg" (UID: "a17239c3-07cb-400b-ac0c-36941de91c01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:37.274238 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.274139    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:35 +0000 UTC" deadline="2028-04-27 02:27:06.355898392 +0000 UTC"
Sep 16 18:45:37.274238 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.274202    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14119h41m29.08170096s"
Sep 16 18:45:37.331395 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.330466    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:37.331395 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:37.330595    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:37.331395 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.330987    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:37.331395 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:37.331076    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:37.331395 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.331142    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:37.331395 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:37.331254    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:37.348577 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.348540    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zbkkp" event={"ID":"e69d4209-41fd-4daf-8d4b-e9fd7f377777","Type":"ContainerStarted","Data":"1d2b3e5b2aa4d7099c974a90bc5c05d0c4f190d10d3ab4b8bad2ee09f5432129"}
Sep 16 18:45:37.353829 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.353721    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg" event={"ID":"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c","Type":"ContainerStarted","Data":"fd0cc1804165fbf85f5807487e1695c0be04529d8acfa076305dd657ae9182b1"}
Sep 16 18:45:37.356920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.356815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fbnt4" event={"ID":"60a41c27-f58c-4070-b3bb-1172234dba44","Type":"ContainerStarted","Data":"93a6f4b5c973bea7d909a897cb4c722b0f540aabd8031af7d847aaa34a486a85"}
Sep 16 18:45:37.359414 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.359333    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wqzk6" event={"ID":"7808bd97-d8b5-43a5-a11f-37dbbf196633","Type":"ContainerStarted","Data":"20d92eab4b4145c6f752e38418eeec9a1b0eb8f6f6d12278b6b1646595913db7"}
Sep 16 18:45:37.367410 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.367351    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs" event={"ID":"af918883-ce09-4d8b-ac4b-d7fb7b190dc4","Type":"ContainerStarted","Data":"0e6ebdc016040a91cd256f4bc5e464b60b79061f248b77a61d89a0816855f00a"}
Sep 16 18:45:37.371475 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.371448    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2z8b" event={"ID":"8856e3f1-9640-4dc7-b3fd-6936fc04f239","Type":"ContainerStarted","Data":"1c29c395f5b261da25f3d06c1fafcf28cf964f023cf6d69e2934628dde57f7bf"}
Sep 16 18:45:37.384721 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.384694    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bhsmz" event={"ID":"ea444e57-cd1a-4620-a3aa-df7e8348c481","Type":"ContainerStarted","Data":"1e76e0db4fb7aa65a69db3172e88fa032b69a4a830fb109388a5242061c48729"}
Sep 16 18:45:37.395261 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.395232    2567 generic.go:356] "Generic (PLEG): container finished" podID="ccd97510a049e6f9ef06e726eab9ef77" containerID="9b6454dd32a283aa7db8f389993404c1846e62fabffb6e7a0fe0adc33ff5b852" exitCode=0
Sep 16 18:45:37.395372 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.395333    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" event={"ID":"ccd97510a049e6f9ef06e726eab9ef77","Type":"ContainerDied","Data":"9b6454dd32a283aa7db8f389993404c1846e62fabffb6e7a0fe0adc33ff5b852"}
Sep 16 18:45:37.405469 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.405437    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qxvlt" event={"ID":"a858b275-e882-46fc-b2c4-164767aaea80","Type":"ContainerStarted","Data":"bfa9831e19910e1762c591ec5e967631bfe0e19ce1d3944396d34ac43788b14a"}
Sep 16 18:45:37.413058 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.413029    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfxbv" event={"ID":"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298","Type":"ContainerStarted","Data":"eca67b726eea5e6df0bfab949a1ff4d48a987f30bfe5a18d83641a8252643e16"}
Sep 16 18:45:37.417766 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:37.417739    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" event={"ID":"cd53a7a796c4569b9de0a17cd4101459","Type":"ContainerStarted","Data":"3b51c50b9194df6ea0deb2ced330f03264aa6a07260709a30c515dd89a6bbd54"}
Sep 16 18:45:38.446407 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:38.446358    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" event={"ID":"ccd97510a049e6f9ef06e726eab9ef77","Type":"ContainerStarted","Data":"7e170b7b80519ac979d16dad829275920682ffaa7a47c09171df343b372cc8ea"}
Sep 16 18:45:38.466315 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:38.465879    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" podStartSLOduration=3.465857072 podStartE2EDuration="3.465857072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:37.442243086 +0000 UTC m=+4.630689769" watchObservedRunningTime="2026-09-16 18:45:38.465857072 +0000 UTC m=+5.654303759"
Sep 16 18:45:38.466315 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:38.466126    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" podStartSLOduration=3.466121062 podStartE2EDuration="3.466121062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:38.465426786 +0000 UTC m=+5.653873472" watchObservedRunningTime="2026-09-16 18:45:38.466121062 +0000 UTC m=+5.654567750"
Sep 16 18:45:38.948251 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:38.942949    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:38.948251 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:38.943099    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:38.948251 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:38.944164    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:38.948251 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:38.944275    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret podName:a2104af8-40ea-48de-8403-6c112903ca6d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:42.94425245 +0000 UTC m=+10.132699118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret") pod "global-pull-secret-syncer-8g5df" (UID: "a2104af8-40ea-48de-8403-6c112903ca6d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:38.949948 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:38.948793    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:38.949948 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:38.948876    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs podName:e7fa29a5-39f2-485f-b1fe-268334ec4935 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:42.948853299 +0000 UTC m=+10.137299969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs") pod "network-metrics-daemon-fsv9w" (UID: "e7fa29a5-39f2-485f-b1fe-268334ec4935") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:39.044802 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:39.044758    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp6m\" (UniqueName: \"kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m\") pod \"network-check-target-g8wrg\" (UID: \"a17239c3-07cb-400b-ac0c-36941de91c01\") " pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:39.045017 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:39.044996    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:39.045136 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:39.045029    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:39.045136 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:39.045044    2567 projected.go:196] Error preparing data for projected volume kube-api-access-jvp6m for pod openshift-network-diagnostics/network-check-target-g8wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:39.045136 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:39.045111    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m podName:a17239c3-07cb-400b-ac0c-36941de91c01 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:43.045092052 +0000 UTC m=+10.233538720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvp6m" (UniqueName: "kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m") pod "network-check-target-g8wrg" (UID: "a17239c3-07cb-400b-ac0c-36941de91c01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:39.329472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:39.329382    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:39.329639 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:39.329516    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:39.329968 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:39.329932    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:39.330099 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:39.330027    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:39.332004 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:39.331979    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:39.332110 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:39.332089    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:41.328088 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:41.328050    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:41.328088 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:41.328064    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:41.328088 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:41.328050    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:41.328760 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:41.328200    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:41.328760 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:41.328268    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:41.328760 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:41.328365    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:42.978993 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:42.978732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:42.979472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:42.979012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:42.979472 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:42.978905    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:42.979472 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:42.979222    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs podName:e7fa29a5-39f2-485f-b1fe-268334ec4935 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.979203501 +0000 UTC m=+18.167650175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs") pod "network-metrics-daemon-fsv9w" (UID: "e7fa29a5-39f2-485f-b1fe-268334ec4935") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:42.979633 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:42.979158    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:42.979633 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:42.979584    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret podName:a2104af8-40ea-48de-8403-6c112903ca6d nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.979566778 +0000 UTC m=+18.168013451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret") pod "global-pull-secret-syncer-8g5df" (UID: "a2104af8-40ea-48de-8403-6c112903ca6d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:43.079727 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:43.079687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp6m\" (UniqueName: \"kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m\") pod \"network-check-target-g8wrg\" (UID: \"a17239c3-07cb-400b-ac0c-36941de91c01\") " pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:43.079906 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:43.079880    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:43.079906 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:43.079904    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:43.080020 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:43.079918    2567 projected.go:196] Error preparing data for projected volume kube-api-access-jvp6m for pod openshift-network-diagnostics/network-check-target-g8wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:43.080020 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:43.079982    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m podName:a17239c3-07cb-400b-ac0c-36941de91c01 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:51.079960876 +0000 UTC m=+18.268407539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvp6m" (UniqueName: "kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m") pod "network-check-target-g8wrg" (UID: "a17239c3-07cb-400b-ac0c-36941de91c01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:43.330530 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:43.329827    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:43.330530 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:43.329934    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:43.330530 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:43.330177    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:43.330530 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:43.330287    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:43.330530 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:43.330315    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:43.330530 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:43.330416    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:45.328257 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:45.328076    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:45.328257 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:45.328218    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:45.328850 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:45.328274    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:45.328850 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:45.328388    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:45.328850 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:45.328419    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:45.328850 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:45.328507    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:47.328217 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:47.328177    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:47.328666 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:47.328177    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:47.328666 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:47.328299    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:47.328666 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:47.328396    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:47.328666 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:47.328177    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:47.328666 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:47.328506    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:49.328193 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:49.328154    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:49.328695 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:49.328176    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:49.328695 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:49.328286    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:49.328695 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:49.328299    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:49.328695 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:49.328384    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:49.328695 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:49.328470    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:51.034269 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:51.034224    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:51.034783 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:51.034297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:51.034783 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.034402    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:51.034783 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.034447    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:51.034783 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.034483    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs podName:e7fa29a5-39f2-485f-b1fe-268334ec4935 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.034464039 +0000 UTC m=+34.222910703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs") pod "network-metrics-daemon-fsv9w" (UID: "e7fa29a5-39f2-485f-b1fe-268334ec4935") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:51.034783 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.034500    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret podName:a2104af8-40ea-48de-8403-6c112903ca6d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.03449372 +0000 UTC m=+34.222940383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret") pod "global-pull-secret-syncer-8g5df" (UID: "a2104af8-40ea-48de-8403-6c112903ca6d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:51.135224 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:51.135186    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp6m\" (UniqueName: \"kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m\") pod \"network-check-target-g8wrg\" (UID: \"a17239c3-07cb-400b-ac0c-36941de91c01\") " pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:51.135376 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.135337    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:51.135376 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.135360    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:51.135376 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.135369    2567 projected.go:196] Error preparing data for projected volume kube-api-access-jvp6m for pod openshift-network-diagnostics/network-check-target-g8wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:51.135530 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.135426    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m podName:a17239c3-07cb-400b-ac0c-36941de91c01 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.13540641 +0000 UTC m=+34.323853070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvp6m" (UniqueName: "kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m") pod "network-check-target-g8wrg" (UID: "a17239c3-07cb-400b-ac0c-36941de91c01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:51.328440 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:51.328357    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:51.328440 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:51.328385    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:51.328633 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.328476    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:51.328633 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.328515    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:51.328633 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:51.328558    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:51.328798 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:51.328637    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:53.329526 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:53.329491    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:53.329896 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:53.329592    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:53.329896 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:53.329601    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:53.329896 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:53.329703    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:53.329896 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:53.329729    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:53.329896 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:53.329806    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:54.495605 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:54.495143    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wqzk6" event={"ID":"7808bd97-d8b5-43a5-a11f-37dbbf196633","Type":"ContainerStarted","Data":"200d0a236ba48d08d32eb40df893163fc79cdcfc806ae3cdb3a5758bc7540cd2"}
Sep 16 18:45:55.329222 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.328956    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:55.329398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.328956    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:55.329398 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:55.329270    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:55.329587 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:55.329413    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:55.329587 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.329037    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:55.329695 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:55.329573    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:55.498565 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.498531    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zbkkp" event={"ID":"e69d4209-41fd-4daf-8d4b-e9fd7f377777","Type":"ContainerStarted","Data":"ba8cd8d29d09ca1f13057457ae3236bb2cc7df7cad7913f6d87f78165e373922"}
Sep 16 18:45:55.502122 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.502083    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg" event={"ID":"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c","Type":"ContainerStarted","Data":"cb972a2944159e1b239a3cea225b2a7c3dba064e390e16571d11e3dd1ee50259"}
Sep 16 18:45:55.502239 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.502197    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg" event={"ID":"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c","Type":"ContainerStarted","Data":"9a698417c25b15531421ed14d61a9cba13d3b8dfc7761579db54ec431edab7ee"}
Sep 16 18:45:55.502296 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.502251    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg" event={"ID":"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c","Type":"ContainerStarted","Data":"1921bc03b2a281c62d0ec1112391e1b8e32a372923657d97fa7a30efd738d8af"}
Sep 16 18:45:55.502351 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.502299    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg" event={"ID":"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c","Type":"ContainerStarted","Data":"40e6cf47cc9a3c15501a3ee443392aaf7af7150c1fbab981ac363b713d518f04"}
Sep 16 18:45:55.502351 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.502347    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg" event={"ID":"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c","Type":"ContainerStarted","Data":"9570a3075e8eb260ebf66f9ea7f9ab83726e9d08eee4c9cf7550bd5f035af0aa"}
Sep 16 18:45:55.502436 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.502376    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg" event={"ID":"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c","Type":"ContainerStarted","Data":"a23a58f0f4522f28830a05124dba58216c765b6015bedc7b1efc65a379e8f667"}
Sep 16 18:45:55.503743 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.503722    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs" event={"ID":"af918883-ce09-4d8b-ac4b-d7fb7b190dc4","Type":"ContainerStarted","Data":"6a8f4d97e13b07aa26fa2a6d69730e2b1c0bd670e1fa7604fbfecdfdd2e9b05c"}
Sep 16 18:45:55.505356 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.505331    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2z8b" event={"ID":"8856e3f1-9640-4dc7-b3fd-6936fc04f239","Type":"ContainerStarted","Data":"a87c917ffcc5955073dbbfe2da1845b8b98150c00ed2a03d89e13eb07b51ba03"}
Sep 16 18:45:55.507004 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.506984    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bhsmz" event={"ID":"ea444e57-cd1a-4620-a3aa-df7e8348c481","Type":"ContainerStarted","Data":"a4bcf6d3a89fbce1db07bc638e06ea1ac64498e210739aa5c5b1698cd6814149"}
Sep 16 18:45:55.508628 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.508604    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qxvlt" event={"ID":"a858b275-e882-46fc-b2c4-164767aaea80","Type":"ContainerStarted","Data":"42ad2ebd97a7aa71734c1b99edf1ae5adbb3ad71982e526471a8d11d20ed3370"}
Sep 16 18:45:55.510298 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.510274    2567 generic.go:356] "Generic (PLEG): container finished" podID="9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298" containerID="1db69f8a6e5c61905b3bf81ec12155fba29e10ee430d48168152a812c71d5808" exitCode=0
Sep 16 18:45:55.510468 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.510362    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfxbv" event={"ID":"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298","Type":"ContainerDied","Data":"1db69f8a6e5c61905b3bf81ec12155fba29e10ee430d48168152a812c71d5808"}
Sep 16 18:45:55.537906 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.537858    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zbkkp" podStartSLOduration=4.643142462 podStartE2EDuration="22.537840773s" totalImagesPullingTime="17.894698311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:55.537496092 +0000 UTC m=+22.725942807" watchObservedRunningTime="2026-09-16 18:45:55.537840773 +0000 UTC m=+22.726287455"
Sep 16 18:45:55.538226 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.538200    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wqzk6" podStartSLOduration=8.85389138 podStartE2EDuration="22.538192586s" totalImagesPullingTime="13.684301206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:54.524869316 +0000 UTC m=+21.713315999" watchObservedRunningTime="2026-09-16 18:45:55.538192586 +0000 UTC m=+22.726639267"
Sep 16 18:45:55.569989 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.569808    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p2z8b" podStartSLOduration=4.521022356 podStartE2EDuration="22.569794881s" totalImagesPullingTime="18.048772525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:55.569408741 +0000 UTC m=+22.757855423" watchObservedRunningTime="2026-09-16 18:45:55.569794881 +0000 UTC m=+22.758241562"
Sep 16 18:45:55.607471 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.607363    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bhsmz" podStartSLOduration=4.726253258 podStartE2EDuration="22.607350086s" totalImagesPullingTime="17.881096828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:55.606965912 +0000 UTC m=+22.795412595" watchObservedRunningTime="2026-09-16 18:45:55.607350086 +0000 UTC m=+22.795796769"
Sep 16 18:45:55.658381 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.658345    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qxvlt" podStartSLOduration=4.752319802 podStartE2EDuration="22.658332707s" totalImagesPullingTime="17.906012905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:55.657665663 +0000 UTC m=+22.846112344" watchObservedRunningTime="2026-09-16 18:45:55.658332707 +0000 UTC m=+22.846779432"
Sep 16 18:45:55.853249 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:55.853226    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:45:56.264516 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:56.264416    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:55.853245382Z","UUID":"4fa5f31d-3c68-4d3f-af72-f83d041b05f4","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:45:56.265753 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:56.265733    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:45:56.265869 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:56.265762    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:45:56.515302 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:56.515212    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fbnt4" event={"ID":"60a41c27-f58c-4070-b3bb-1172234dba44","Type":"ContainerStarted","Data":"bad074858b505249290b15d45340c6e14c882ce60034658cfa830f4467eb3b40"}
Sep 16 18:45:56.518313 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:56.518284    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs" event={"ID":"af918883-ce09-4d8b-ac4b-d7fb7b190dc4","Type":"ContainerStarted","Data":"a19bf355e83a9273c4ff9009ab7d376ca94f39a81dbb5fa0aba3cf2addee0a0e"}
Sep 16 18:45:56.536522 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:56.536471    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fbnt4" podStartSLOduration=4.639086398 podStartE2EDuration="22.536457551s" totalImagesPullingTime="17.897371153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:56.535728625 +0000 UTC m=+23.724175314" watchObservedRunningTime="2026-09-16 18:45:56.536457551 +0000 UTC m=+23.724904232"
Sep 16 18:45:57.328292 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:57.328218    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:57.328292 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:57.328232    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:57.328487 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:57.328218    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:57.328487 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:57.328324    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:57.328487 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:57.328419    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:45:57.328582 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:57.328492    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:57.524818 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:57.524779    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg" event={"ID":"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c","Type":"ContainerStarted","Data":"c0c732beba81232fb203567ed048c51c4ee4ebdf50b59a96873aae4abe558ba8"}
Sep 16 18:45:57.527439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:57.527397    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs" event={"ID":"af918883-ce09-4d8b-ac4b-d7fb7b190dc4","Type":"ContainerStarted","Data":"c4646634fbb2dcccd2115c48f7dfa0942717c5c07acba12e1e6ac1668dca35c5"}
Sep 16 18:45:57.551487 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:57.551437    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfsvs" podStartSLOduration=4.358766943 podStartE2EDuration="24.551421997s" totalImagesPullingTime="20.192655054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:57.549634562 +0000 UTC m=+24.738081244" watchObservedRunningTime="2026-09-16 18:45:57.551421997 +0000 UTC m=+24.739868679"
Sep 16 18:45:57.774014 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:57.773984    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:57.774571 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:57.774550    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:58.530238 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:58.530206    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:58.530748 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:58.530582    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wqzk6"
Sep 16 18:45:59.328041 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:59.328013    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:45:59.328222 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:59.328013    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:45:59.328222 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:59.328159    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:45:59.328324 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:59.328226    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:45:59.328324 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:45:59.328253    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:45:59.328394 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:45:59.328379    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:46:00.536261 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:00.536069    2567 generic.go:356] "Generic (PLEG): container finished" podID="9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298" containerID="f038ac0df7d06945f83da6ebdae744002684157fc11c9f8f6e84c05365521041" exitCode=0
Sep 16 18:46:00.536818 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:00.536140    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfxbv" event={"ID":"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298","Type":"ContainerDied","Data":"f038ac0df7d06945f83da6ebdae744002684157fc11c9f8f6e84c05365521041"}
Sep 16 18:46:00.540261 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:00.540239    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg" event={"ID":"6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c","Type":"ContainerStarted","Data":"2104b09a93b84b4f9ec43de649dc2cbfa24eef3614a1e796f3fc8b968482a591"}
Sep 16 18:46:00.540479 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:00.540456    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:46:00.540586 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:00.540508    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:46:00.555310 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:00.555293    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:46:00.596491 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:00.596447    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg" podStartSLOduration=9.691139167 podStartE2EDuration="27.596435106s" totalImagesPullingTime="17.905295939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:00.59482429 +0000 UTC m=+27.783270972" watchObservedRunningTime="2026-09-16 18:46:00.596435106 +0000 UTC m=+27.784881820"
Sep 16 18:46:01.329074 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.329045    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:46:01.329217 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.329105    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:46:01.329253 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:01.329218    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:46:01.329288 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:01.329242    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:46:01.329288 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.329104    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:46:01.329344 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:01.329330    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:46:01.545913 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.545881    2567 generic.go:356] "Generic (PLEG): container finished" podID="9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298" containerID="fffa815ffe83ba89e46a9d8493d10d5cd0901572646d89000f1ab9e19d3a8fda" exitCode=0
Sep 16 18:46:01.546429 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.545974    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfxbv" event={"ID":"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298","Type":"ContainerDied","Data":"fffa815ffe83ba89e46a9d8493d10d5cd0901572646d89000f1ab9e19d3a8fda"}
Sep 16 18:46:01.546561 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.546540    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:46:01.562920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.562901    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:46:01.786844 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.786676    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g8wrg"]
Sep 16 18:46:01.786964 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.786907    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:46:01.787016 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:01.786992    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:46:01.788183 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.788161    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8g5df"]
Sep 16 18:46:01.788269 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.788232    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:46:01.788308 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:01.788296    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:46:01.805911 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.805829    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fsv9w"]
Sep 16 18:46:01.806054 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:01.805915    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:46:01.806054 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:01.806007    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:46:02.550278 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:02.550193    2567 generic.go:356] "Generic (PLEG): container finished" podID="9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298" containerID="8bce63323f1b39e787160c9bef5fb7570ff0e0afabe9102f0fb06bdd2736ed2b" exitCode=0
Sep 16 18:46:02.550652 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:02.550270    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfxbv" event={"ID":"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298","Type":"ContainerDied","Data":"8bce63323f1b39e787160c9bef5fb7570ff0e0afabe9102f0fb06bdd2736ed2b"}
Sep 16 18:46:03.329531 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:03.329162    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:46:03.329531 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:03.329188    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:46:03.329531 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:03.329270    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:46:03.329531 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:03.329333    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:46:03.329531 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:03.329369    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:46:03.329531 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:03.329472    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:46:05.328863 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:05.328835    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:46:05.328863 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:05.328852    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:46:05.329416 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:05.328870    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:46:05.329416 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:05.328940    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8wrg" podUID="a17239c3-07cb-400b-ac0c-36941de91c01"
Sep 16 18:46:05.329416 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:05.329003    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fsv9w" podUID="e7fa29a5-39f2-485f-b1fe-268334ec4935"
Sep 16 18:46:05.329416 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:05.329062    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8g5df" podUID="a2104af8-40ea-48de-8403-6c112903ca6d"
Sep 16 18:46:07.044765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.044702    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeReady"
Sep 16 18:46:07.045182 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.044853    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:46:07.048919 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.048899    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:46:07.049008 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.048938    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:46:07.049045 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.049036    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:07.049077 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.049037    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:07.049122 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.049080    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret podName:a2104af8-40ea-48de-8403-6c112903ca6d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:39.049068414 +0000 UTC m=+66.237515074 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret") pod "global-pull-secret-syncer-8g5df" (UID: "a2104af8-40ea-48de-8403-6c112903ca6d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:07.049122 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.049094    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs podName:e7fa29a5-39f2-485f-b1fe-268334ec4935 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:39.049087081 +0000 UTC m=+66.237533740 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs") pod "network-metrics-daemon-fsv9w" (UID: "e7fa29a5-39f2-485f-b1fe-268334ec4935") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:07.096287 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.096255    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mf48l"]
Sep 16 18:46:07.112151 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.112098    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gv2gr"]
Sep 16 18:46:07.112301 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.112246    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.115134 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.115097    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:46:07.115333 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.115315    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:46:07.115676 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.115659    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k8559\""
Sep 16 18:46:07.124485 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.124468    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gv2gr"]
Sep 16 18:46:07.124549 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.124495    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mf48l"]
Sep 16 18:46:07.124586 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.124564    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gv2gr"
Sep 16 18:46:07.127604 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.127584    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dts4n\""
Sep 16 18:46:07.127709 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.127619    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:46:07.127709 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.127627    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:46:07.127709 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.127589    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:46:07.149203 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.149182    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b545178-c81e-4173-9675-50aaa359f497-tmp-dir\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.149312 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.149217    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.149312 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.149235    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b545178-c81e-4173-9675-50aaa359f497-config-volume\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.149312 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.149297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp6m\" (UniqueName: \"kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m\") pod \"network-check-target-g8wrg\" (UID: \"a17239c3-07cb-400b-ac0c-36941de91c01\") " pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:46:07.149436 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.149337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqll\" (UniqueName: \"kubernetes.io/projected/3b545178-c81e-4173-9675-50aaa359f497-kube-api-access-gjqll\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.149436 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.149412    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:46:07.149436 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.149432    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:46:07.149573 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.149444    2567 projected.go:196] Error preparing data for projected volume kube-api-access-jvp6m for pod openshift-network-diagnostics/network-check-target-g8wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:07.149573 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.149498    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m podName:a17239c3-07cb-400b-ac0c-36941de91c01 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:39.149482385 +0000 UTC m=+66.337929046 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvp6m" (UniqueName: "kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m") pod "network-check-target-g8wrg" (UID: "a17239c3-07cb-400b-ac0c-36941de91c01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:07.250421 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.250387    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b545178-c81e-4173-9675-50aaa359f497-tmp-dir\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.250599 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.250435    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.250599 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.250460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b545178-c81e-4173-9675-50aaa359f497-config-volume\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.250599 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.250519    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqll\" (UniqueName: \"kubernetes.io/projected/3b545178-c81e-4173-9675-50aaa359f497-kube-api-access-gjqll\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.250758 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.250598    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:07.250758 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.250640    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnnr6\" (UniqueName: \"kubernetes.io/projected/9dc205a3-9cc1-4f77-9b00-8df61cf69ba4-kube-api-access-dnnr6\") pod \"ingress-canary-gv2gr\" (UID: \"9dc205a3-9cc1-4f77-9b00-8df61cf69ba4\") " pod="openshift-ingress-canary/ingress-canary-gv2gr"
Sep 16 18:46:07.250758 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.250659    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls podName:3b545178-c81e-4173-9675-50aaa359f497 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.750643016 +0000 UTC m=+34.939089676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls") pod "dns-default-mf48l" (UID: "3b545178-c81e-4173-9675-50aaa359f497") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:07.250758 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.250694    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dc205a3-9cc1-4f77-9b00-8df61cf69ba4-cert\") pod \"ingress-canary-gv2gr\" (UID: \"9dc205a3-9cc1-4f77-9b00-8df61cf69ba4\") " pod="openshift-ingress-canary/ingress-canary-gv2gr"
Sep 16 18:46:07.250969 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.250951    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b545178-c81e-4173-9675-50aaa359f497-tmp-dir\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.251174 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.251160    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b545178-c81e-4173-9675-50aaa359f497-config-volume\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.262433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.262407    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqll\" (UniqueName: \"kubernetes.io/projected/3b545178-c81e-4173-9675-50aaa359f497-kube-api-access-gjqll\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.328042 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.328020    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:46:07.328161 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.328028    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:46:07.328273 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.328254    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:46:07.331546 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.331523    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtjvb\""
Sep 16 18:46:07.331546 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.331530    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:07.331700 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.331621    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:07.331700 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.331624    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqjhk\""
Sep 16 18:46:07.331807 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.331643    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:07.331807 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.331752    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:07.350958 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.350938    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnnr6\" (UniqueName: \"kubernetes.io/projected/9dc205a3-9cc1-4f77-9b00-8df61cf69ba4-kube-api-access-dnnr6\") pod \"ingress-canary-gv2gr\" (UID: \"9dc205a3-9cc1-4f77-9b00-8df61cf69ba4\") " pod="openshift-ingress-canary/ingress-canary-gv2gr"
Sep 16 18:46:07.351037 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.350964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dc205a3-9cc1-4f77-9b00-8df61cf69ba4-cert\") pod \"ingress-canary-gv2gr\" (UID: \"9dc205a3-9cc1-4f77-9b00-8df61cf69ba4\") " pod="openshift-ingress-canary/ingress-canary-gv2gr"
Sep 16 18:46:07.353143 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.353125    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dc205a3-9cc1-4f77-9b00-8df61cf69ba4-cert\") pod \"ingress-canary-gv2gr\" (UID: \"9dc205a3-9cc1-4f77-9b00-8df61cf69ba4\") " pod="openshift-ingress-canary/ingress-canary-gv2gr"
Sep 16 18:46:07.360162 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.360139    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnnr6\" (UniqueName: \"kubernetes.io/projected/9dc205a3-9cc1-4f77-9b00-8df61cf69ba4-kube-api-access-dnnr6\") pod \"ingress-canary-gv2gr\" (UID: \"9dc205a3-9cc1-4f77-9b00-8df61cf69ba4\") " pod="openshift-ingress-canary/ingress-canary-gv2gr"
Sep 16 18:46:07.432195 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.432168    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gv2gr"
Sep 16 18:46:07.563758 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.563697    2567 generic.go:356] "Generic (PLEG): container finished" podID="9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298" containerID="2b08bba6a6df7c14d8b223f0360762c48639fafdb589b7d5df1f2e7d187c2621" exitCode=0
Sep 16 18:46:07.563758 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.563753    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfxbv" event={"ID":"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298","Type":"ContainerDied","Data":"2b08bba6a6df7c14d8b223f0360762c48639fafdb589b7d5df1f2e7d187c2621"}
Sep 16 18:46:07.754354 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.754327    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:07.754499 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.754480    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:07.754550 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:07.754541    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls podName:3b545178-c81e-4173-9675-50aaa359f497 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:08.754526105 +0000 UTC m=+35.942972765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls") pod "dns-default-mf48l" (UID: "3b545178-c81e-4173-9675-50aaa359f497") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:07.767394 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:07.767076    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gv2gr"]
Sep 16 18:46:07.783881 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:46:07.783855    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc205a3_9cc1_4f77_9b00_8df61cf69ba4.slice/crio-6db138ffbd171e969e3562406769ee5dcb3870a9ebf39e0b9b8dbad1c54e3088 WatchSource:0}: Error finding container 6db138ffbd171e969e3562406769ee5dcb3870a9ebf39e0b9b8dbad1c54e3088: Status 404 returned error can't find the container with id 6db138ffbd171e969e3562406769ee5dcb3870a9ebf39e0b9b8dbad1c54e3088
Sep 16 18:46:08.569938 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:08.569908    2567 generic.go:356] "Generic (PLEG): container finished" podID="9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298" containerID="65e27ea8c3ed081678cf484c0c669ba5262482f3ae34c2a71106229441b56083" exitCode=0
Sep 16 18:46:08.570754 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:08.569981    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfxbv" event={"ID":"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298","Type":"ContainerDied","Data":"65e27ea8c3ed081678cf484c0c669ba5262482f3ae34c2a71106229441b56083"}
Sep 16 18:46:08.571410 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:08.571355    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gv2gr" event={"ID":"9dc205a3-9cc1-4f77-9b00-8df61cf69ba4","Type":"ContainerStarted","Data":"6db138ffbd171e969e3562406769ee5dcb3870a9ebf39e0b9b8dbad1c54e3088"}
Sep 16 18:46:08.763262 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:08.763041    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:08.763366 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:08.763223    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:08.763438 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:08.763386    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls podName:3b545178-c81e-4173-9675-50aaa359f497 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:10.763364972 +0000 UTC m=+37.951811683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls") pod "dns-default-mf48l" (UID: "3b545178-c81e-4173-9675-50aaa359f497") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:09.577309 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:09.577281    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfxbv" event={"ID":"9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298","Type":"ContainerStarted","Data":"448a42a61e354b9ad0c886c873f7477a46498703ed175c20fa5c96d2d90b3f28"}
Sep 16 18:46:09.607457 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:09.607412    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vfxbv" podStartSLOduration=8.830178651 podStartE2EDuration="36.607394808s" totalImagesPullingTime="27.777216157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:09.605181856 +0000 UTC m=+36.793628538" watchObservedRunningTime="2026-09-16 18:46:09.607394808 +0000 UTC m=+36.795841486"
Sep 16 18:46:10.581728 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:10.581647    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gv2gr" event={"ID":"9dc205a3-9cc1-4f77-9b00-8df61cf69ba4","Type":"ContainerStarted","Data":"7b2d656722dcaefe8f2b29c08f3a5ba061a98cd5dddafb293fc3d1b9fa6a95f4"}
Sep 16 18:46:10.611401 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:10.611343    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gv2gr" podStartSLOduration=1.07617668 podStartE2EDuration="3.61132318s" totalImagesPullingTime="2.5351465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:10.609420306 +0000 UTC m=+37.797866988" watchObservedRunningTime="2026-09-16 18:46:10.61132318 +0000 UTC m=+37.799769864"
Sep 16 18:46:10.777660 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:10.777627    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:10.777823 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:10.777755    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:10.777823 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:10.777815    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls podName:3b545178-c81e-4173-9675-50aaa359f497 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:14.777798047 +0000 UTC m=+41.966244706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls") pod "dns-default-mf48l" (UID: "3b545178-c81e-4173-9675-50aaa359f497") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:14.802402 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:14.802353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:14.802867 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:14.802491    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:14.802867 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:14.802572    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls podName:3b545178-c81e-4173-9675-50aaa359f497 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:22.802552429 +0000 UTC m=+49.990999107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls") pod "dns-default-mf48l" (UID: "3b545178-c81e-4173-9675-50aaa359f497") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:22.856229 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:22.856191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:22.856608 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:22.856310    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:22.856608 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:22.856384    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls podName:3b545178-c81e-4173-9675-50aaa359f497 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:38.856370261 +0000 UTC m=+66.044816920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls") pod "dns-default-mf48l" (UID: "3b545178-c81e-4173-9675-50aaa359f497") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:33.563431 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:33.563404    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69rtg"
Sep 16 18:46:38.863458 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:38.863423    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:46:38.863910 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:38.863532    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:38.863910 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:38.863586    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls podName:3b545178-c81e-4173-9675-50aaa359f497 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.863570874 +0000 UTC m=+98.052017537 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls") pod "dns-default-mf48l" (UID: "3b545178-c81e-4173-9675-50aaa359f497") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:39.065491 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.065450    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:46:39.065654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.065524    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:46:39.068902 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.068884    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:39.068954 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.068915    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:39.076502 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:39.076483    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:46:39.076581 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:39.076562    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs podName:e7fa29a5-39f2-485f-b1fe-268334ec4935 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:43.076540354 +0000 UTC m=+130.264987014 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs") pod "network-metrics-daemon-fsv9w" (UID: "e7fa29a5-39f2-485f-b1fe-268334ec4935") : secret "metrics-daemon-secret" not found
Sep 16 18:46:39.079546 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.079525    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2104af8-40ea-48de-8403-6c112903ca6d-original-pull-secret\") pod \"global-pull-secret-syncer-8g5df\" (UID: \"a2104af8-40ea-48de-8403-6c112903ca6d\") " pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:46:39.147244 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.147210    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8g5df"
Sep 16 18:46:39.166179 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.166152    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp6m\" (UniqueName: \"kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m\") pod \"network-check-target-g8wrg\" (UID: \"a17239c3-07cb-400b-ac0c-36941de91c01\") " pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:46:39.169353 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.169334    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:39.179493 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.179476    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:39.190053 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.190031    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvp6m\" (UniqueName: \"kubernetes.io/projected/a17239c3-07cb-400b-ac0c-36941de91c01-kube-api-access-jvp6m\") pod \"network-check-target-g8wrg\" (UID: \"a17239c3-07cb-400b-ac0c-36941de91c01\") " pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:46:39.416312 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.416247    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8g5df"]
Sep 16 18:46:39.419421 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:46:39.419393    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2104af8_40ea_48de_8403_6c112903ca6d.slice/crio-f1aee8f881f1b4df2f76532b571c38c3aa71e7055035a57b86edecc711a3f1b4 WatchSource:0}: Error finding container f1aee8f881f1b4df2f76532b571c38c3aa71e7055035a57b86edecc711a3f1b4: Status 404 returned error can't find the container with id f1aee8f881f1b4df2f76532b571c38c3aa71e7055035a57b86edecc711a3f1b4
Sep 16 18:46:39.440669 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.440646    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqjhk\""
Sep 16 18:46:39.448317 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.448296    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:46:39.649096 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.649064    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8g5df" event={"ID":"a2104af8-40ea-48de-8403-6c112903ca6d","Type":"ContainerStarted","Data":"f1aee8f881f1b4df2f76532b571c38c3aa71e7055035a57b86edecc711a3f1b4"}
Sep 16 18:46:39.716028 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:39.715967    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g8wrg"]
Sep 16 18:46:39.718698 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:46:39.718677    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17239c3_07cb_400b_ac0c_36941de91c01.slice/crio-137095eea63314b3e1745479fbb367ec319941ebd5ceb8ef84b5f7f14208def8 WatchSource:0}: Error finding container 137095eea63314b3e1745479fbb367ec319941ebd5ceb8ef84b5f7f14208def8: Status 404 returned error can't find the container with id 137095eea63314b3e1745479fbb367ec319941ebd5ceb8ef84b5f7f14208def8
Sep 16 18:46:40.656082 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:40.656038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g8wrg" event={"ID":"a17239c3-07cb-400b-ac0c-36941de91c01","Type":"ContainerStarted","Data":"137095eea63314b3e1745479fbb367ec319941ebd5ceb8ef84b5f7f14208def8"}
Sep 16 18:46:44.668862 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:44.668828    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8g5df" event={"ID":"a2104af8-40ea-48de-8403-6c112903ca6d","Type":"ContainerStarted","Data":"67ce8539ba4acc13858ead44f5e154c2c100b102e6dde41ddfe68805ba9eda5d"}
Sep 16 18:46:44.670336 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:44.670312    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g8wrg" event={"ID":"a17239c3-07cb-400b-ac0c-36941de91c01","Type":"ContainerStarted","Data":"67082210ab03be2ab3a32eba0adba9fd37ac00064c6e365f3344bdffeaf04303"}
Sep 16 18:46:44.670496 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:44.670477    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:46:44.687383 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:44.687340    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8g5df" podStartSLOduration=66.237487878 podStartE2EDuration="1m10.687327535s" totalImagesPullingTime="4.449839657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:44.686686781 +0000 UTC m=+71.875133465" watchObservedRunningTime="2026-09-16 18:46:44.687327535 +0000 UTC m=+71.875774217"
Sep 16 18:46:57.737246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.737098    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g8wrg" podStartSLOduration=80.588068192 podStartE2EDuration="1m24.737082744s" totalImagesPullingTime="4.149014552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:44.705033193 +0000 UTC m=+71.893479876" watchObservedRunningTime="2026-09-16 18:46:57.737082744 +0000 UTC m=+84.925529493"
Sep 16 18:46:57.737637 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.737542    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59"]
Sep 16 18:46:57.739698 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.739683    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59"
Sep 16 18:46:57.742460 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.742436    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:57.743573 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.743553    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2k4tl\""
Sep 16 18:46:57.743660 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.743624    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:57.750817 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.750796    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59"]
Sep 16 18:46:57.836570 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.836537    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"]
Sep 16 18:46:57.838342 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.838327    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:46:57.839733 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.839712    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-2jhcw"]
Sep 16 18:46:57.840837 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.840822    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rbxxh\""
Sep 16 18:46:57.840885 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.840849    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:46:57.841118 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.841091    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:57.841172 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.841100    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:57.842032 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.842016    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:57.844638 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.844617    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:46:57.844733 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.844676    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:46:57.844733 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.844619    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d45dx\""
Sep 16 18:46:57.845037 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.845018    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:46:57.845600 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.845580    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:46:57.852325 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.852307    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:46:57.852725 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.852709    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"]
Sep 16 18:46:57.854401 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.854384    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-2jhcw"]
Sep 16 18:46:57.887057 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.887030    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2m4\" (UniqueName: \"kubernetes.io/projected/3c6ef1b4-3a09-40c2-bc93-d99b1e8962b8-kube-api-access-7k2m4\") pod \"volume-data-source-validator-86695b7cc7-l7d59\" (UID: \"3c6ef1b4-3a09-40c2-bc93-d99b1e8962b8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59"
Sep 16 18:46:57.943239 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.943212    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7khsl"]
Sep 16 18:46:57.944966 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.944947    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:57.948228 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.948212    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:57.948308 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.948212    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:46:57.948557 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.948537    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:46:57.948643 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.948538    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:57.948743 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.948717    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pbhv2\""
Sep 16 18:46:57.953947 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.953926    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:46:57.959843 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.959823    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7khsl"]
Sep 16 18:46:57.987352 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.987283    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81bd4b21-6b45-4504-b47f-a6201451b987-serving-cert\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:57.987352 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.987322    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81bd4b21-6b45-4504-b47f-a6201451b987-snapshots\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:57.987491 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.987387    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bd4b21-6b45-4504-b47f-a6201451b987-tmp\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:57.987491 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.987407    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmvg\" (UniqueName: \"kubernetes.io/projected/5278b4f9-768e-43a2-960a-a1683664243b-kube-api-access-zvmvg\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:46:57.987491 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.987430    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2m4\" (UniqueName: \"kubernetes.io/projected/3c6ef1b4-3a09-40c2-bc93-d99b1e8962b8-kube-api-access-7k2m4\") pod \"volume-data-source-validator-86695b7cc7-l7d59\" (UID: \"3c6ef1b4-3a09-40c2-bc93-d99b1e8962b8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59"
Sep 16 18:46:57.987491 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.987446    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81bd4b21-6b45-4504-b47f-a6201451b987-service-ca-bundle\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:57.987630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.987491    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpm2d\" (UniqueName: \"kubernetes.io/projected/81bd4b21-6b45-4504-b47f-a6201451b987-kube-api-access-hpm2d\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:57.987630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.987551    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81bd4b21-6b45-4504-b47f-a6201451b987-trusted-ca-bundle\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:57.987630 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.987571    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:46:57.999858 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:57.999836    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2m4\" (UniqueName: \"kubernetes.io/projected/3c6ef1b4-3a09-40c2-bc93-d99b1e8962b8-kube-api-access-7k2m4\") pod \"volume-data-source-validator-86695b7cc7-l7d59\" (UID: \"3c6ef1b4-3a09-40c2-bc93-d99b1e8962b8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59"
Sep 16 18:46:58.047536 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.047500    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59"
Sep 16 18:46:58.088386 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088350    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmvg\" (UniqueName: \"kubernetes.io/projected/5278b4f9-768e-43a2-960a-a1683664243b-kube-api-access-zvmvg\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:46:58.088525 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c74sd\" (UniqueName: \"kubernetes.io/projected/007217df-062a-4583-8425-a5f6df57af08-kube-api-access-c74sd\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.088525 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088467    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81bd4b21-6b45-4504-b47f-a6201451b987-service-ca-bundle\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.088525 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpm2d\" (UniqueName: \"kubernetes.io/projected/81bd4b21-6b45-4504-b47f-a6201451b987-kube-api-access-hpm2d\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.088525 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088522    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/007217df-062a-4583-8425-a5f6df57af08-trusted-ca\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.088711 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81bd4b21-6b45-4504-b47f-a6201451b987-trusted-ca-bundle\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.088711 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088576    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:46:58.088711 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81bd4b21-6b45-4504-b47f-a6201451b987-serving-cert\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.088856 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088717    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/007217df-062a-4583-8425-a5f6df57af08-config\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.088856 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:58.088672    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:46:58.088856 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088759    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81bd4b21-6b45-4504-b47f-a6201451b987-snapshots\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.088856 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:58.088799    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls podName:5278b4f9-768e-43a2-960a-a1683664243b nodeName:}" failed. No retries permitted until 2026-09-16 18:46:58.588776416 +0000 UTC m=+85.777223101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tjwc5" (UID: "5278b4f9-768e-43a2-960a-a1683664243b") : secret "samples-operator-tls" not found
Sep 16 18:46:58.088856 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088826    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/007217df-062a-4583-8425-a5f6df57af08-serving-cert\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.089037 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.088882    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bd4b21-6b45-4504-b47f-a6201451b987-tmp\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.089124 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.089091    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81bd4b21-6b45-4504-b47f-a6201451b987-service-ca-bundle\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.089384 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.089358    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bd4b21-6b45-4504-b47f-a6201451b987-tmp\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.089497 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.089442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81bd4b21-6b45-4504-b47f-a6201451b987-trusted-ca-bundle\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.089570 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.089552    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81bd4b21-6b45-4504-b47f-a6201451b987-snapshots\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.091185 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.091143    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81bd4b21-6b45-4504-b47f-a6201451b987-serving-cert\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.099554 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.099529    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpm2d\" (UniqueName: \"kubernetes.io/projected/81bd4b21-6b45-4504-b47f-a6201451b987-kube-api-access-hpm2d\") pod \"insights-operator-5844b856db-2jhcw\" (UID: \"81bd4b21-6b45-4504-b47f-a6201451b987\") " pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.099697 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.099559    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmvg\" (UniqueName: \"kubernetes.io/projected/5278b4f9-768e-43a2-960a-a1683664243b-kube-api-access-zvmvg\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:46:58.153625 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.153596    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-2jhcw"
Sep 16 18:46:58.189577 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.189547    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/007217df-062a-4583-8425-a5f6df57af08-config\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.189577 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.189582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/007217df-062a-4583-8425-a5f6df57af08-serving-cert\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.189820 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.189704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c74sd\" (UniqueName: \"kubernetes.io/projected/007217df-062a-4583-8425-a5f6df57af08-kube-api-access-c74sd\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.189820 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.189727    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/007217df-062a-4583-8425-a5f6df57af08-trusted-ca\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.190392 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.190366    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/007217df-062a-4583-8425-a5f6df57af08-config\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.190567 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.190550    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/007217df-062a-4583-8425-a5f6df57af08-trusted-ca\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.191975 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.191958    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/007217df-062a-4583-8425-a5f6df57af08-serving-cert\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.200264 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.200001    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c74sd\" (UniqueName: \"kubernetes.io/projected/007217df-062a-4583-8425-a5f6df57af08-kube-api-access-c74sd\") pod \"console-operator-696884fb9c-7khsl\" (UID: \"007217df-062a-4583-8425-a5f6df57af08\") " pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.253898 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.253835    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:46:58.318580 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.318552    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59"]
Sep 16 18:46:58.322123 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:46:58.322082    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6ef1b4_3a09_40c2_bc93_d99b1e8962b8.slice/crio-622a54b996b61e9569f029c16fb860b2aba55ae6142b92becde9f1722c877cbf WatchSource:0}: Error finding container 622a54b996b61e9569f029c16fb860b2aba55ae6142b92becde9f1722c877cbf: Status 404 returned error can't find the container with id 622a54b996b61e9569f029c16fb860b2aba55ae6142b92becde9f1722c877cbf
Sep 16 18:46:58.426220 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.424767    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-2jhcw"]
Sep 16 18:46:58.525221 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.525155    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7khsl"]
Sep 16 18:46:58.528665 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:46:58.528637    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007217df_062a_4583_8425_a5f6df57af08.slice/crio-50467b460be44c8437a4474822e558824eae56c72a3185214f0ce86fd8532fbe WatchSource:0}: Error finding container 50467b460be44c8437a4474822e558824eae56c72a3185214f0ce86fd8532fbe: Status 404 returned error can't find the container with id 50467b460be44c8437a4474822e558824eae56c72a3185214f0ce86fd8532fbe
Sep 16 18:46:58.593447 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.593419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:46:58.593564 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:58.593528    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:46:58.593602 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:58.593576    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls podName:5278b4f9-768e-43a2-960a-a1683664243b nodeName:}" failed. No retries permitted until 2026-09-16 18:46:59.593563238 +0000 UTC m=+86.782009898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tjwc5" (UID: "5278b4f9-768e-43a2-960a-a1683664243b") : secret "samples-operator-tls" not found
Sep 16 18:46:58.705940 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.705901    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-2jhcw" event={"ID":"81bd4b21-6b45-4504-b47f-a6201451b987","Type":"ContainerStarted","Data":"4687093a24cba081f3a697749882110a7f2f5462938d607032f940bdd1fc17a0"}
Sep 16 18:46:58.707149 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.707121    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59" event={"ID":"3c6ef1b4-3a09-40c2-bc93-d99b1e8962b8","Type":"ContainerStarted","Data":"622a54b996b61e9569f029c16fb860b2aba55ae6142b92becde9f1722c877cbf"}
Sep 16 18:46:58.708169 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:58.708146    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7khsl" event={"ID":"007217df-062a-4583-8425-a5f6df57af08","Type":"ContainerStarted","Data":"50467b460be44c8437a4474822e558824eae56c72a3185214f0ce86fd8532fbe"}
Sep 16 18:46:59.601935 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:59.601851    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:46:59.602373 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:59.602004    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:46:59.602373 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:46:59.602074    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls podName:5278b4f9-768e-43a2-960a-a1683664243b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:01.602055764 +0000 UTC m=+88.790502429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tjwc5" (UID: "5278b4f9-768e-43a2-960a-a1683664243b") : secret "samples-operator-tls" not found
Sep 16 18:46:59.713056 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:59.713017    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59" event={"ID":"3c6ef1b4-3a09-40c2-bc93-d99b1e8962b8","Type":"ContainerStarted","Data":"1e1f2b6a1cb6cf9a1e2836a1d63b5c63fb23fa732242afd0798a5ff989a7455e"}
Sep 16 18:46:59.734231 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:46:59.734172    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l7d59" podStartSLOduration=1.717783047 podStartE2EDuration="2.734151134s" totalImagesPullingTime="1.016368087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:59.733517928 +0000 UTC m=+86.921964612" watchObservedRunningTime="2026-09-16 18:46:59.734151134 +0000 UTC m=+86.922597817"
Sep 16 18:47:00.036257 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.036219    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56dc97c7df-vcth6"]
Sep 16 18:47:00.039248 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.039220    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.042705 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.042678    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:47:00.042879 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.042712    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-htp7b\""
Sep 16 18:47:00.043638 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.043618    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:47:00.043638 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.043633    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:47:00.054389 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.053673    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:47:00.054946 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.054922    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56dc97c7df-vcth6"]
Sep 16 18:47:00.206102 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.206060    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.206102 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.206093    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d974f52-86bb-420f-a91d-39f2dfa2c62b-ca-trust-extracted\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.206352 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.206176    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-image-registry-private-configuration\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.206352 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.206206    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-installation-pull-secrets\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.206352 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.206277    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-certificates\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.206352 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.206296    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-trusted-ca\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.206352 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.206313    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-bound-sa-token\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.206352 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.206347    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp8mv\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-kube-api-access-fp8mv\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.307208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.307132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp8mv\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-kube-api-access-fp8mv\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.307208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.307174    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.307208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.307197    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d974f52-86bb-420f-a91d-39f2dfa2c62b-ca-trust-extracted\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.307472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.307235    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-image-registry-private-configuration\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.307472 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:00.307321    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:00.307472 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:00.307346    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56dc97c7df-vcth6: secret "image-registry-tls" not found
Sep 16 18:47:00.307472 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:00.307407    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls podName:4d974f52-86bb-420f-a91d-39f2dfa2c62b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:00.807386224 +0000 UTC m=+87.995832891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls") pod "image-registry-56dc97c7df-vcth6" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b") : secret "image-registry-tls" not found
Sep 16 18:47:00.307472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.307430    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-installation-pull-secrets\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.307725 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.307527    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-certificates\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.307725 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.307557    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-trusted-ca\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.307725 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.307580    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-bound-sa-token\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.307725 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.307617    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d974f52-86bb-420f-a91d-39f2dfa2c62b-ca-trust-extracted\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.308160 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.308137    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-certificates\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.308522 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.308496    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-trusted-ca\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.309910 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.309890    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-image-registry-private-configuration\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.310005 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.309962    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-installation-pull-secrets\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.325524 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.325494    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp8mv\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-kube-api-access-fp8mv\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.325649 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.325630    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-bound-sa-token\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.810367 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:00.810337    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:00.810734 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:00.810484    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:00.810734 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:00.810506    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56dc97c7df-vcth6: secret "image-registry-tls" not found
Sep 16 18:47:00.810734 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:00.810555    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls podName:4d974f52-86bb-420f-a91d-39f2dfa2c62b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:01.81053958 +0000 UTC m=+88.998986240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls") pod "image-registry-56dc97c7df-vcth6" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b") : secret "image-registry-tls" not found
Sep 16 18:47:01.614342 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:01.614307    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:47:01.614501 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:01.614465    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:47:01.614546 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:01.614533    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls podName:5278b4f9-768e-43a2-960a-a1683664243b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:05.614515477 +0000 UTC m=+92.802962136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tjwc5" (UID: "5278b4f9-768e-43a2-960a-a1683664243b") : secret "samples-operator-tls" not found
Sep 16 18:47:01.720865 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:01.720837    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/0.log"
Sep 16 18:47:01.721031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:01.720876    2567 generic.go:356] "Generic (PLEG): container finished" podID="007217df-062a-4583-8425-a5f6df57af08" containerID="0a3abf82bc1c10b106d6a2a5a7853a8ebe1c33de16425f5f03c99b9b25be21f8" exitCode=255
Sep 16 18:47:01.721031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:01.720952    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7khsl" event={"ID":"007217df-062a-4583-8425-a5f6df57af08","Type":"ContainerDied","Data":"0a3abf82bc1c10b106d6a2a5a7853a8ebe1c33de16425f5f03c99b9b25be21f8"}
Sep 16 18:47:01.721203 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:01.721186    2567 scope.go:117] "RemoveContainer" containerID="0a3abf82bc1c10b106d6a2a5a7853a8ebe1c33de16425f5f03c99b9b25be21f8"
Sep 16 18:47:01.722555 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:01.722528    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-2jhcw" event={"ID":"81bd4b21-6b45-4504-b47f-a6201451b987","Type":"ContainerStarted","Data":"0cf06ef482fa453ff11ad734896a2491875a30d62d129848ae1ebe7c70664f46"}
Sep 16 18:47:01.759898 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:01.759852    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-2jhcw" podStartSLOduration=2.467112721 podStartE2EDuration="4.759834082s" totalImagesPullingTime="2.292721361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.759267933 +0000 UTC m=+88.947714615" watchObservedRunningTime="2026-09-16 18:47:01.759834082 +0000 UTC m=+88.948280768"
Sep 16 18:47:01.816319 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:01.816240    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:01.816755 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:01.816365    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:01.816755 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:01.816381    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56dc97c7df-vcth6: secret "image-registry-tls" not found
Sep 16 18:47:01.816755 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:01.816448    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls podName:4d974f52-86bb-420f-a91d-39f2dfa2c62b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:03.816425675 +0000 UTC m=+91.004872349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls") pod "image-registry-56dc97c7df-vcth6" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b") : secret "image-registry-tls" not found
Sep 16 18:47:02.727515 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.727490    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 18:47:02.727938 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.727922    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/0.log"
Sep 16 18:47:02.727993 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.727957    2567 generic.go:356] "Generic (PLEG): container finished" podID="007217df-062a-4583-8425-a5f6df57af08" containerID="53c7a2d309f1c2c2b2a96e2d94ff373f8d870919a28ce48cab13d5262f925b2e" exitCode=255
Sep 16 18:47:02.728043 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.728030    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7khsl" event={"ID":"007217df-062a-4583-8425-a5f6df57af08","Type":"ContainerDied","Data":"53c7a2d309f1c2c2b2a96e2d94ff373f8d870919a28ce48cab13d5262f925b2e"}
Sep 16 18:47:02.728082 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.728074    2567 scope.go:117] "RemoveContainer" containerID="0a3abf82bc1c10b106d6a2a5a7853a8ebe1c33de16425f5f03c99b9b25be21f8"
Sep 16 18:47:02.728328 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.728301    2567 scope.go:117] "RemoveContainer" containerID="53c7a2d309f1c2c2b2a96e2d94ff373f8d870919a28ce48cab13d5262f925b2e"
Sep 16 18:47:02.728494 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:02.728474    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7khsl_openshift-console-operator(007217df-062a-4583-8425-a5f6df57af08)\"" pod="openshift-console-operator/console-operator-696884fb9c-7khsl" podUID="007217df-062a-4583-8425-a5f6df57af08"
Sep 16 18:47:02.746104 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.746082    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wkl5v"]
Sep 16 18:47:02.749039 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.749024    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:02.752646 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.752627    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:47:02.752806 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.752790    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:47:02.752847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.752829    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-46pc5\""
Sep 16 18:47:02.766446 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.766424    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkl5v"]
Sep 16 18:47:02.923944 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.923917    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74280712-1f17-494c-a01e-e0c17d385c96-data-volume\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:02.924336 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.923961    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74280712-1f17-494c-a01e-e0c17d385c96-crio-socket\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:02.924336 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.923985    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:02.924336 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.924048    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4mj\" (UniqueName: \"kubernetes.io/projected/74280712-1f17-494c-a01e-e0c17d385c96-kube-api-access-fk4mj\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:02.924336 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:02.924101    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.025233 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.025154    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74280712-1f17-494c-a01e-e0c17d385c96-data-volume\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.025233 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.025204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74280712-1f17-494c-a01e-e0c17d385c96-crio-socket\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.025403 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.025238    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.025403 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.025278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4mj\" (UniqueName: \"kubernetes.io/projected/74280712-1f17-494c-a01e-e0c17d385c96-kube-api-access-fk4mj\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.025403 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.025338    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.025403 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.025346    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74280712-1f17-494c-a01e-e0c17d385c96-crio-socket\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.025542 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.025399    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:03.025542 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.025421    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:03.025542 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.025456    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:03.525441861 +0000 UTC m=+90.713888521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:03.025542 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.025478    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:03.525461843 +0000 UTC m=+90.713908524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:03.025542 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.025519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74280712-1f17-494c-a01e-e0c17d385c96-data-volume\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.034188 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.034167    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4mj\" (UniqueName: \"kubernetes.io/projected/74280712-1f17-494c-a01e-e0c17d385c96-kube-api-access-fk4mj\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.529026 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.528992    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.529203 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.529050    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:03.529203 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.529139    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:03.529203 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.529162    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:03.529203 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.529189    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:04.529176422 +0000 UTC m=+91.717623081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:03.529341 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.529216    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:04.529202819 +0000 UTC m=+91.717649479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:03.732663 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.732637    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 18:47:03.732943 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.732928    2567 scope.go:117] "RemoveContainer" containerID="53c7a2d309f1c2c2b2a96e2d94ff373f8d870919a28ce48cab13d5262f925b2e"
Sep 16 18:47:03.733079 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.733063    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7khsl_openshift-console-operator(007217df-062a-4583-8425-a5f6df57af08)\"" pod="openshift-console-operator/console-operator-696884fb9c-7khsl" podUID="007217df-062a-4583-8425-a5f6df57af08"
Sep 16 18:47:03.830779 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:03.830704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:03.830940 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.830840    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:03.830940 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.830871    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56dc97c7df-vcth6: secret "image-registry-tls" not found
Sep 16 18:47:03.831049 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:03.830954    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls podName:4d974f52-86bb-420f-a91d-39f2dfa2c62b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:07.830931503 +0000 UTC m=+95.019378178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls") pod "image-registry-56dc97c7df-vcth6" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b") : secret "image-registry-tls" not found
Sep 16 18:47:04.536236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:04.536186    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:04.536608 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:04.536284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:04.536608 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:04.536336    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:04.536608 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:04.536374    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:04.536608 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:04.536415    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:06.536399669 +0000 UTC m=+93.724846329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:04.536608 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:04.536447    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:06.536440988 +0000 UTC m=+93.724887648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:05.355648 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.355622    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-bnc2v"]
Sep 16 18:47:05.357595 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.357580    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.360531 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.360504    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:47:05.360661 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.360571    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:47:05.360661 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.360585    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:47:05.361872 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.361854    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:47:05.361972 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.361881    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tppjt\""
Sep 16 18:47:05.374686 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.374662    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-bnc2v"]
Sep 16 18:47:05.543884 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.543844    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cd7e2d7-3abb-4120-b675-85da7ebe6d3e-signing-key\") pod \"service-ca-568869fdc5-bnc2v\" (UID: \"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e\") " pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.544275 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.543915    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cd7e2d7-3abb-4120-b675-85da7ebe6d3e-signing-cabundle\") pod \"service-ca-568869fdc5-bnc2v\" (UID: \"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e\") " pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.544275 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.544003    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9sq\" (UniqueName: \"kubernetes.io/projected/0cd7e2d7-3abb-4120-b675-85da7ebe6d3e-kube-api-access-ch9sq\") pod \"service-ca-568869fdc5-bnc2v\" (UID: \"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e\") " pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.621605 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.621582    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bhsmz_ea444e57-cd1a-4620-a3aa-df7e8348c481/dns-node-resolver/0.log"
Sep 16 18:47:05.645059 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.645031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cd7e2d7-3abb-4120-b675-85da7ebe6d3e-signing-cabundle\") pod \"service-ca-568869fdc5-bnc2v\" (UID: \"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e\") " pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.645199 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.645084    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9sq\" (UniqueName: \"kubernetes.io/projected/0cd7e2d7-3abb-4120-b675-85da7ebe6d3e-kube-api-access-ch9sq\") pod \"service-ca-568869fdc5-bnc2v\" (UID: \"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e\") " pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.645199 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.645137    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:47:05.645274 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:05.645236    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:47:05.645316 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:05.645282    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls podName:5278b4f9-768e-43a2-960a-a1683664243b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:13.645270393 +0000 UTC m=+100.833717053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tjwc5" (UID: "5278b4f9-768e-43a2-960a-a1683664243b") : secret "samples-operator-tls" not found
Sep 16 18:47:05.645371 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.645332    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cd7e2d7-3abb-4120-b675-85da7ebe6d3e-signing-key\") pod \"service-ca-568869fdc5-bnc2v\" (UID: \"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e\") " pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.645667 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.645645    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cd7e2d7-3abb-4120-b675-85da7ebe6d3e-signing-cabundle\") pod \"service-ca-568869fdc5-bnc2v\" (UID: \"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e\") " pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.647581 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.647559    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cd7e2d7-3abb-4120-b675-85da7ebe6d3e-signing-key\") pod \"service-ca-568869fdc5-bnc2v\" (UID: \"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e\") " pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.653959 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.653938    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9sq\" (UniqueName: \"kubernetes.io/projected/0cd7e2d7-3abb-4120-b675-85da7ebe6d3e-kube-api-access-ch9sq\") pod \"service-ca-568869fdc5-bnc2v\" (UID: \"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e\") " pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.666907 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.666888    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-bnc2v"
Sep 16 18:47:05.931726 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:05.931659    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-bnc2v"]
Sep 16 18:47:05.934588 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:05.934561    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd7e2d7_3abb_4120_b675_85da7ebe6d3e.slice/crio-99e4dd0de0082891c2834c133fc3bc1a899c48a2a2fb3f3a146067f28911248b WatchSource:0}: Error finding container 99e4dd0de0082891c2834c133fc3bc1a899c48a2a2fb3f3a146067f28911248b: Status 404 returned error can't find the container with id 99e4dd0de0082891c2834c133fc3bc1a899c48a2a2fb3f3a146067f28911248b
Sep 16 18:47:06.554301 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:06.554263    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:06.554742 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:06.554419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:06.554742 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:06.554445    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:06.554742 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:06.554502    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:06.554742 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:06.554539    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.55449858 +0000 UTC m=+97.742945282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:06.554742 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:06.554560    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.554550479 +0000 UTC m=+97.742997144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:06.622388 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:06.622362    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zbkkp_e69d4209-41fd-4daf-8d4b-e9fd7f377777/node-ca/0.log"
Sep 16 18:47:06.744380 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:06.744343    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-bnc2v" event={"ID":"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e","Type":"ContainerStarted","Data":"99e4dd0de0082891c2834c133fc3bc1a899c48a2a2fb3f3a146067f28911248b"}
Sep 16 18:47:07.224370 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:07.224328    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gv2gr_9dc205a3-9cc1-4f77-9b00-8df61cf69ba4/serve-healthcheck-canary/0.log"
Sep 16 18:47:07.749089 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:07.749055    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-bnc2v" event={"ID":"0cd7e2d7-3abb-4120-b675-85da7ebe6d3e","Type":"ContainerStarted","Data":"ea5f37bf85912c0f8483b12f992169d654e8b9b5509c5326a5aeb37150f303d6"}
Sep 16 18:47:07.770667 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:07.770621    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-bnc2v" podStartSLOduration=1.031750022 podStartE2EDuration="2.770606427s" totalImagesPullingTime="1.738856405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:07.770277744 +0000 UTC m=+94.958724425" watchObservedRunningTime="2026-09-16 18:47:07.770606427 +0000 UTC m=+94.959053144"
Sep 16 18:47:07.865125 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:07.865089    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:07.865381 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:07.865348    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:07.865381 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:07.865374    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56dc97c7df-vcth6: secret "image-registry-tls" not found
Sep 16 18:47:07.865546 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:07.865425    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls podName:4d974f52-86bb-420f-a91d-39f2dfa2c62b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:15.865409858 +0000 UTC m=+103.053856537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls") pod "image-registry-56dc97c7df-vcth6" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b") : secret "image-registry-tls" not found
Sep 16 18:47:08.253968 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:08.253918    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:47:08.254148 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:08.253992    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:47:08.254362 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:08.254349    2567 scope.go:117] "RemoveContainer" containerID="53c7a2d309f1c2c2b2a96e2d94ff373f8d870919a28ce48cab13d5262f925b2e"
Sep 16 18:47:08.254513 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:08.254495    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7khsl_openshift-console-operator(007217df-062a-4583-8425-a5f6df57af08)\"" pod="openshift-console-operator/console-operator-696884fb9c-7khsl" podUID="007217df-062a-4583-8425-a5f6df57af08"
Sep 16 18:47:10.586300 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:10.586258    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:10.586660 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:10.586353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:10.586660 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:10.586380    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:10.586660 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:10.586435    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:18.586423104 +0000 UTC m=+105.774869769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:10.586660 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:10.586498    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:10.586660 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:10.586551    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:18.586538067 +0000 UTC m=+105.774984727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:10.889676 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:10.889645    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:47:10.889826 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:10.889797    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:47:10.889863 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:10.889857    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls podName:3b545178-c81e-4173-9675-50aaa359f497 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:14.889842302 +0000 UTC m=+162.078288962 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls") pod "dns-default-mf48l" (UID: "3b545178-c81e-4173-9675-50aaa359f497") : secret "dns-default-metrics-tls" not found
Sep 16 18:47:13.714520 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:13.714478    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:47:13.716921 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:13.716895    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5278b4f9-768e-43a2-960a-a1683664243b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tjwc5\" (UID: \"5278b4f9-768e-43a2-960a-a1683664243b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:47:13.748831 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:13.748800    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"
Sep 16 18:47:14.038727 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:14.038701    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5"]
Sep 16 18:47:14.772659 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:14.772625    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5" event={"ID":"5278b4f9-768e-43a2-960a-a1683664243b","Type":"ContainerStarted","Data":"30b796e4d90e55a37711e9e2aa8196a2cb1519a708ec1428c41f1a2672abb2b9"}
Sep 16 18:47:15.675744 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:15.675714    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g8wrg"
Sep 16 18:47:15.934019 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:15.933924    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:15.936495 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:15.936469    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls\") pod \"image-registry-56dc97c7df-vcth6\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") " pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:15.951240 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:15.951210    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:16.307800 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:16.307769    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56dc97c7df-vcth6"]
Sep 16 18:47:16.311850 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:16.311826    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d974f52_86bb_420f_a91d_39f2dfa2c62b.slice/crio-1057faf161c12c8541f07377a5f94a53ec299f7d1f54097e3c6f081d0e31996e WatchSource:0}: Error finding container 1057faf161c12c8541f07377a5f94a53ec299f7d1f54097e3c6f081d0e31996e: Status 404 returned error can't find the container with id 1057faf161c12c8541f07377a5f94a53ec299f7d1f54097e3c6f081d0e31996e
Sep 16 18:47:16.783906 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:16.783862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5" event={"ID":"5278b4f9-768e-43a2-960a-a1683664243b","Type":"ContainerStarted","Data":"2b70916df12db37423408aaee4cc181164032f8a6fcfe5d5a4d0f1244203e0e8"}
Sep 16 18:47:16.784231 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:16.783926    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5" event={"ID":"5278b4f9-768e-43a2-960a-a1683664243b","Type":"ContainerStarted","Data":"47ec9f9c6993b814dc9b960661f03309abd7e04a55f7c57aa77eaad5c7564dbc"}
Sep 16 18:47:16.785509 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:16.785486    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6" event={"ID":"4d974f52-86bb-420f-a91d-39f2dfa2c62b","Type":"ContainerStarted","Data":"84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c"}
Sep 16 18:47:16.785622 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:16.785525    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6" event={"ID":"4d974f52-86bb-420f-a91d-39f2dfa2c62b","Type":"ContainerStarted","Data":"1057faf161c12c8541f07377a5f94a53ec299f7d1f54097e3c6f081d0e31996e"}
Sep 16 18:47:16.785666 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:16.785616    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:16.811535 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:16.811462    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tjwc5" podStartSLOduration=17.86990099 podStartE2EDuration="19.811439671s" totalImagesPullingTime="1.941538681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:16.809561562 +0000 UTC m=+103.998008244" watchObservedRunningTime="2026-09-16 18:47:16.811439671 +0000 UTC m=+103.999886354"
Sep 16 18:47:16.838122 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:16.838069    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6" podStartSLOduration=16.838049011 podStartE2EDuration="16.838049011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:16.83747845 +0000 UTC m=+104.025925131" watchObservedRunningTime="2026-09-16 18:47:16.838049011 +0000 UTC m=+104.026495695"
Sep 16 18:47:18.656247 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:18.656214    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:18.656614 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:18.656271    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:18.656614 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:18.656369    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:18.656614 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:18.656445    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls podName:74280712-1f17-494c-a01e-e0c17d385c96 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:34.656430329 +0000 UTC m=+121.844876988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkl5v" (UID: "74280712-1f17-494c-a01e-e0c17d385c96") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:18.656788 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:18.656772    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74280712-1f17-494c-a01e-e0c17d385c96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:22.328859 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:22.328831    2567 scope.go:117] "RemoveContainer" containerID="53c7a2d309f1c2c2b2a96e2d94ff373f8d870919a28ce48cab13d5262f925b2e"
Sep 16 18:47:22.805380 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:22.805354    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 18:47:22.805606 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:22.805440    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7khsl" event={"ID":"007217df-062a-4583-8425-a5f6df57af08","Type":"ContainerStarted","Data":"cb17d8576b90b5b5fa00d7b160770dc2aa92579689a3f70e116990e93fabaf81"}
Sep 16 18:47:22.805793 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:22.805775    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:47:22.829192 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:22.829138    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-7khsl" podStartSLOduration=23.636775824 podStartE2EDuration="25.829105266s" totalImagesPullingTime="2.192329442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:22.827361608 +0000 UTC m=+110.015808290" watchObservedRunningTime="2026-09-16 18:47:22.829105266 +0000 UTC m=+110.017551949"
Sep 16 18:47:22.887904 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:22.887881    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-7khsl"
Sep 16 18:47:29.189996 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.189961    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"]
Sep 16 18:47:29.192048 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.192028    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"
Sep 16 18:47:29.195702 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.195681    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tbv7l\""
Sep 16 18:47:29.197062 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.197045    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:47:29.197196 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.197171    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:47:29.218095 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.218075    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"]
Sep 16 18:47:29.229932 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.229906    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d95f7bc-6499-4c74-9c04-41a9b501f6ba-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-4gg9x\" (UID: \"7d95f7bc-6499-4c74-9c04-41a9b501f6ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"
Sep 16 18:47:29.230038 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.229961    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d95f7bc-6499-4c74-9c04-41a9b501f6ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4gg9x\" (UID: \"7d95f7bc-6499-4c74-9c04-41a9b501f6ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"
Sep 16 18:47:29.237093 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.237070    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56dc97c7df-vcth6"]
Sep 16 18:47:29.330394 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.330361    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d95f7bc-6499-4c74-9c04-41a9b501f6ba-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-4gg9x\" (UID: \"7d95f7bc-6499-4c74-9c04-41a9b501f6ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"
Sep 16 18:47:29.330550 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.330413    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d95f7bc-6499-4c74-9c04-41a9b501f6ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4gg9x\" (UID: \"7d95f7bc-6499-4c74-9c04-41a9b501f6ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"
Sep 16 18:47:29.331034 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.331010    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d95f7bc-6499-4c74-9c04-41a9b501f6ba-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-4gg9x\" (UID: \"7d95f7bc-6499-4c74-9c04-41a9b501f6ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"
Sep 16 18:47:29.332899 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.332876    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d95f7bc-6499-4c74-9c04-41a9b501f6ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4gg9x\" (UID: \"7d95f7bc-6499-4c74-9c04-41a9b501f6ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"
Sep 16 18:47:29.501087 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.500998    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"
Sep 16 18:47:29.789098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.789028    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x"]
Sep 16 18:47:29.791810 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:29.791783    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d95f7bc_6499_4c74_9c04_41a9b501f6ba.slice/crio-a54039e96ea57a34bd963b314c11bfab6836193f0f70d6d54d4ae98e46c0cf35 WatchSource:0}: Error finding container a54039e96ea57a34bd963b314c11bfab6836193f0f70d6d54d4ae98e46c0cf35: Status 404 returned error can't find the container with id a54039e96ea57a34bd963b314c11bfab6836193f0f70d6d54d4ae98e46c0cf35
Sep 16 18:47:29.826905 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:29.826870    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x" event={"ID":"7d95f7bc-6499-4c74-9c04-41a9b501f6ba","Type":"ContainerStarted","Data":"a54039e96ea57a34bd963b314c11bfab6836193f0f70d6d54d4ae98e46c0cf35"}
Sep 16 18:47:31.835994 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:31.835961    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x" event={"ID":"7d95f7bc-6499-4c74-9c04-41a9b501f6ba","Type":"ContainerStarted","Data":"1f0b88232785f40e7722304a5313b7b63090f11633e4c04299c646285da5c4df"}
Sep 16 18:47:31.854529 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:31.854481    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4gg9x" podStartSLOduration=1.801459451 podStartE2EDuration="2.85446523s" totalImagesPullingTime="1.053005779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:31.853910409 +0000 UTC m=+119.042357085" watchObservedRunningTime="2026-09-16 18:47:31.85446523 +0000 UTC m=+119.042911945"
Sep 16 18:47:32.826238 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:32.826200    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"]
Sep 16 18:47:32.829427 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:32.829406    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"
Sep 16 18:47:32.832448 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:32.832429    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:47:32.832806 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:32.832789    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v25tx\""
Sep 16 18:47:32.837915 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:32.837843    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"]
Sep 16 18:47:32.855236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:32.855212    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hgmkg\" (UID: \"85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"
Sep 16 18:47:32.955732 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:32.955694    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hgmkg\" (UID: \"85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"
Sep 16 18:47:32.955882 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:32.955828    2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:47:32.955931 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:32.955901    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3-tls-certificates podName:85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:33.455879687 +0000 UTC m=+120.644326369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-hgmkg" (UID: "85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:47:33.460522 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.460472    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hgmkg\" (UID: \"85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"
Sep 16 18:47:33.462887 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.462853    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hgmkg\" (UID: \"85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"
Sep 16 18:47:33.708714 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.708675    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c886bf65c-7xr2b"]
Sep 16 18:47:33.711818 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.711747    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.715519 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.715497    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:47:33.715625 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.715497    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:47:33.716277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.716258    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4p59w\""
Sep 16 18:47:33.716803 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.716780    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:47:33.716915 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.716805    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:47:33.716915 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.716838    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:47:33.717013 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.716950    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:47:33.720503 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.720487    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:47:33.736713 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.736688    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c886bf65c-7xr2b"]
Sep 16 18:47:33.741600 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.741574    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v25tx\""
Sep 16 18:47:33.748914 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.748896    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"
Sep 16 18:47:33.763284 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.763260    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-config\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.763369 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.763306    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-serving-cert\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.763369 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.763337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-oauth-config\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.763467 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.763449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsmfs\" (UniqueName: \"kubernetes.io/projected/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-kube-api-access-fsmfs\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.763508 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.763493    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-service-ca\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.763554 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.763519    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-oauth-serving-cert\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.864466 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.864437    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsmfs\" (UniqueName: \"kubernetes.io/projected/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-kube-api-access-fsmfs\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.864815 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.864483    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-service-ca\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.864815 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.864507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-oauth-serving-cert\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.864815 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.864527    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-config\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.864815 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.864591    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-serving-cert\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.864815 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.864629    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-oauth-config\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.865226 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.865208    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-config\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.865344 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.865325    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-service-ca\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.865404 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.865374    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-oauth-serving-cert\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.867024 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.867004    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-oauth-config\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.867136 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.867024    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-serving-cert\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:33.876654 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:33.876628    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsmfs\" (UniqueName: \"kubernetes.io/projected/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-kube-api-access-fsmfs\") pod \"console-c886bf65c-7xr2b\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") " pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:34.020867 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:34.020795    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:34.026184 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:34.026157    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"]
Sep 16 18:47:34.029682 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:34.029660    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b5a4ec_d7ab_4ad9_9cf6_e1b3c0a271c3.slice/crio-14fdced5ef7f5f6a0c22f53b910325cfb16cb5c4f27b673f6c12496358f3b57f WatchSource:0}: Error finding container 14fdced5ef7f5f6a0c22f53b910325cfb16cb5c4f27b673f6c12496358f3b57f: Status 404 returned error can't find the container with id 14fdced5ef7f5f6a0c22f53b910325cfb16cb5c4f27b673f6c12496358f3b57f
Sep 16 18:47:34.301890 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:34.301819    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c886bf65c-7xr2b"]
Sep 16 18:47:34.304908 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:34.304885    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49eb2ae_8b1e_4fdd_b15a_f783cc63c3f8.slice/crio-d878857228b9609b8bc067dfbd49fd6519a8c9fe6885713f1cbcce5059af355f WatchSource:0}: Error finding container d878857228b9609b8bc067dfbd49fd6519a8c9fe6885713f1cbcce5059af355f: Status 404 returned error can't find the container with id d878857228b9609b8bc067dfbd49fd6519a8c9fe6885713f1cbcce5059af355f
Sep 16 18:47:34.670492 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:34.670452    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:34.673188 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:34.673159    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74280712-1f17-494c-a01e-e0c17d385c96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl5v\" (UID: \"74280712-1f17-494c-a01e-e0c17d385c96\") " pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:34.845581 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:34.845547    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c886bf65c-7xr2b" event={"ID":"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8","Type":"ContainerStarted","Data":"d878857228b9609b8bc067dfbd49fd6519a8c9fe6885713f1cbcce5059af355f"}
Sep 16 18:47:34.846562 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:34.846539    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg" event={"ID":"85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3","Type":"ContainerStarted","Data":"14fdced5ef7f5f6a0c22f53b910325cfb16cb5c4f27b673f6c12496358f3b57f"}
Sep 16 18:47:34.864524 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:34.861244    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-46pc5\""
Sep 16 18:47:34.867821 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:34.867803    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkl5v"
Sep 16 18:47:35.147928 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.147892    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkl5v"]
Sep 16 18:47:35.150816 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:35.150789    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74280712_1f17_494c_a01e_e0c17d385c96.slice/crio-cb70976088cc3a7c528e5b61367f54755b6a7bfb52347a73d1000b337c0ece34 WatchSource:0}: Error finding container cb70976088cc3a7c528e5b61367f54755b6a7bfb52347a73d1000b337c0ece34: Status 404 returned error can't find the container with id cb70976088cc3a7c528e5b61367f54755b6a7bfb52347a73d1000b337c0ece34
Sep 16 18:47:35.834241 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.834210    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65799dbbd-v4tfv"]
Sep 16 18:47:35.836668 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.836626    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.851434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.851409    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:47:35.852966 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.852937    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg" event={"ID":"85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3","Type":"ContainerStarted","Data":"b557223a68e4b5fcfbffcc2fda2bb8c5788c2bf5a31b2d5be6221b39253f0805"}
Sep 16 18:47:35.853087 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.852992    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"
Sep 16 18:47:35.855461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.855370    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkl5v" event={"ID":"74280712-1f17-494c-a01e-e0c17d385c96","Type":"ContainerStarted","Data":"7dbfd6e4c1b0392ed02dc5be77280c4b492fc7b5be9ed918c6fc8dc60df609aa"}
Sep 16 18:47:35.855461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.855425    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkl5v" event={"ID":"74280712-1f17-494c-a01e-e0c17d385c96","Type":"ContainerStarted","Data":"cb70976088cc3a7c528e5b61367f54755b6a7bfb52347a73d1000b337c0ece34"}
Sep 16 18:47:35.859274 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.858990    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65799dbbd-v4tfv"]
Sep 16 18:47:35.859437 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.859418    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg"
Sep 16 18:47:35.881810 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.881778    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-trusted-ca-bundle\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.882161 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.881835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-config\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.882161 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.881898    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-service-ca\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.882161 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.881989    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-oauth-config\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.882161 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.882025    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-oauth-serving-cert\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.882161 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.882056    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-serving-cert\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.882161 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.882085    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvhk\" (UniqueName: \"kubernetes.io/projected/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-kube-api-access-nvvhk\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.912702 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.912643    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hgmkg" podStartSLOduration=2.953375247 podStartE2EDuration="3.912620483s" totalImagesPullingTime="959.245236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:35.911922515 +0000 UTC m=+123.100369198" watchObservedRunningTime="2026-09-16 18:47:35.912620483 +0000 UTC m=+123.101067168"
Sep 16 18:47:35.982931 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.982860    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-config\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.983049 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.982966    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-service-ca\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.983249 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.983208    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-oauth-config\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.983343 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.983280    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-oauth-serving-cert\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.983426 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.983338    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-serving-cert\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.983426 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.983368    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvhk\" (UniqueName: \"kubernetes.io/projected/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-kube-api-access-nvvhk\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.983514 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.983435    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-trusted-ca-bundle\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.983897 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.983568    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-config\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.983897 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.983717    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-service-ca\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.984258 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.984222    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-oauth-serving-cert\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.984722 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.984454    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-trusted-ca-bundle\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.986787 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.986745    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-serving-cert\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.986886 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.986846    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-oauth-config\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:35.996028 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:35.996000    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvhk\" (UniqueName: \"kubernetes.io/projected/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-kube-api-access-nvvhk\") pod \"console-65799dbbd-v4tfv\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") " pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:36.148486 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:36.148436    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:36.429786 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:36.429697    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65799dbbd-v4tfv"]
Sep 16 18:47:36.862073 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:36.861999    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkl5v" event={"ID":"74280712-1f17-494c-a01e-e0c17d385c96","Type":"ContainerStarted","Data":"db41ada9c7f102742cfdcfa7191b888d5a5cb1517fd523e3593070fc42310f4b"}
Sep 16 18:47:37.264443 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:37.264385    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8d8757_4193_4971_a87a_2dc7b8ba30fe.slice/crio-db8c60a45ab97443d9bc1bb5e333ddc29ddcf4a96c2a2542b8071f4b35f13b0e WatchSource:0}: Error finding container db8c60a45ab97443d9bc1bb5e333ddc29ddcf4a96c2a2542b8071f4b35f13b0e: Status 404 returned error can't find the container with id db8c60a45ab97443d9bc1bb5e333ddc29ddcf4a96c2a2542b8071f4b35f13b0e
Sep 16 18:47:37.870937 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:37.870887    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c886bf65c-7xr2b" event={"ID":"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8","Type":"ContainerStarted","Data":"019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632"}
Sep 16 18:47:37.872926 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:37.872900    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65799dbbd-v4tfv" event={"ID":"0a8d8757-4193-4971-a87a-2dc7b8ba30fe","Type":"ContainerStarted","Data":"8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3"}
Sep 16 18:47:37.873068 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:37.872944    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65799dbbd-v4tfv" event={"ID":"0a8d8757-4193-4971-a87a-2dc7b8ba30fe","Type":"ContainerStarted","Data":"db8c60a45ab97443d9bc1bb5e333ddc29ddcf4a96c2a2542b8071f4b35f13b0e"}
Sep 16 18:47:37.898577 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:37.898520    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c886bf65c-7xr2b" podStartSLOduration=1.9095835270000001 podStartE2EDuration="4.898501731s" totalImagesPullingTime="2.988918204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:37.897195047 +0000 UTC m=+125.085641729" watchObservedRunningTime="2026-09-16 18:47:37.898501731 +0000 UTC m=+125.086948413"
Sep 16 18:47:37.926125 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:37.926011    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65799dbbd-v4tfv" podStartSLOduration=2.520520046 podStartE2EDuration="2.925995731s" totalImagesPullingTime="405.475685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:37.925119882 +0000 UTC m=+125.113566557" watchObservedRunningTime="2026-09-16 18:47:37.925995731 +0000 UTC m=+125.114442412"
Sep 16 18:47:38.878598 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:38.878560    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkl5v" event={"ID":"74280712-1f17-494c-a01e-e0c17d385c96","Type":"ContainerStarted","Data":"c39dc203acc58204c431e98baf30ac0c202497025d2a59fdd520a67d97585b04"}
Sep 16 18:47:38.906345 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:38.906297    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wkl5v" podStartSLOduration=34.111921831 podStartE2EDuration="36.90627938s" totalImagesPullingTime="2.794357549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:38.904405024 +0000 UTC m=+126.092851702" watchObservedRunningTime="2026-09-16 18:47:38.90627938 +0000 UTC m=+126.094726060"
Sep 16 18:47:39.242378 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:39.242303    2567 patch_prober.go:28] interesting pod/image-registry-56dc97c7df-vcth6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:47:39.242378 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:39.242363    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6" podUID="4d974f52-86bb-420f-a91d-39f2dfa2c62b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:47:41.602736 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.602699    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"]
Sep 16 18:47:41.606105 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.606083    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"]
Sep 16 18:47:41.606250 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.606221    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.608491 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.608470    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.622402 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.622383    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:47:41.622550 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.622394    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:47:41.622988 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.622968    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-s5r9w\""
Sep 16 18:47:41.623206 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.623185    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:47:41.623449 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.623197    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:47:41.623609 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:41.623486    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"openshift-state-metrics-tls\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret"
Sep 16 18:47:41.623609 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:41.623486    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"openshift-state-metrics-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret"
Sep 16 18:47:41.623609 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.623556    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:47:41.623609 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.623579    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:47:41.623821 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:41.623627    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"openshift-state-metrics-dockercfg-zb5dl\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zb5dl\"" type="*v1.Secret"
Sep 16 18:47:41.630181 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.630160    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5b246"]
Sep 16 18:47:41.632552 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.632535    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.645836 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.645811    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:47:41.646658 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.646643    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:47:41.646748 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.646658    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:47:41.646748 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.646645    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vbsp9\""
Sep 16 18:47:41.657412 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.657386    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"]
Sep 16 18:47:41.728337 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728309    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qncv\" (UniqueName: \"kubernetes.io/projected/d8c25ba8-9217-4dcc-b645-00ca89c6641e-kube-api-access-2qncv\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.728456 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728367    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.728456 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728397    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.728456 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728430    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/49e3d9c7-c939-45a4-acc1-940afde69201-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.728608 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728456    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.728608 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728519    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-tls\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.728608 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728562    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.728738 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728614    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.728738 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728642    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z4qq\" (UniqueName: \"kubernetes.io/projected/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-kube-api-access-6z4qq\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.728738 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728667    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-wtmp\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.728738 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728707    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.728991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728751    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8c25ba8-9217-4dcc-b645-00ca89c6641e-sys\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.728991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728778    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8c25ba8-9217-4dcc-b645-00ca89c6641e-metrics-client-ca\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.728991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728803    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.728991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728866    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.728991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728892    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxtz6\" (UniqueName: \"kubernetes.io/projected/49e3d9c7-c939-45a4-acc1-940afde69201-kube-api-access-kxtz6\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.728991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728916    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8c25ba8-9217-4dcc-b645-00ca89c6641e-root\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.728991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728938    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e3d9c7-c939-45a4-acc1-940afde69201-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.728991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.728969    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-textfile\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.733211 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.733190    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"]
Sep 16 18:47:41.829766 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.829732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.829963 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.829771    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.829963 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.829801    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z4qq\" (UniqueName: \"kubernetes.io/projected/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-kube-api-access-6z4qq\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.829963 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.829825    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-wtmp\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.829963 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:41.829902    2567 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 18:47:41.830208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.829966    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-wtmp\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.829987    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.830208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830059    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8c25ba8-9217-4dcc-b645-00ca89c6641e-sys\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830208 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:41.830090    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-tls podName:49e3d9c7-c939-45a4-acc1-940afde69201 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:42.330067922 +0000 UTC m=+129.518514583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-5qgmh" (UID: "49e3d9c7-c939-45a4-acc1-940afde69201") : secret "kube-state-metrics-tls" not found
Sep 16 18:47:41.830208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830118    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8c25ba8-9217-4dcc-b645-00ca89c6641e-sys\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830146    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8c25ba8-9217-4dcc-b645-00ca89c6641e-metrics-client-ca\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.830208 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830206    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830232    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxtz6\" (UniqueName: \"kubernetes.io/projected/49e3d9c7-c939-45a4-acc1-940afde69201-kube-api-access-kxtz6\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830261    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8c25ba8-9217-4dcc-b645-00ca89c6641e-root\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830337    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e3d9c7-c939-45a4-acc1-940afde69201-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830382    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-textfile\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830441    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qncv\" (UniqueName: \"kubernetes.io/projected/d8c25ba8-9217-4dcc-b645-00ca89c6641e-kube-api-access-2qncv\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830488    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830515    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830556    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/49e3d9c7-c939-45a4-acc1-940afde69201-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830583    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830623    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:41.830712    2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830721    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8c25ba8-9217-4dcc-b645-00ca89c6641e-metrics-client-ca\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:41.830769    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-tls podName:d8c25ba8-9217-4dcc-b645-00ca89c6641e nodeName:}" failed. No retries permitted until 2026-09-16 18:47:42.33075358 +0000 UTC m=+129.519200254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-tls") pod "node-exporter-5b246" (UID: "d8c25ba8-9217-4dcc-b645-00ca89c6641e") : secret "node-exporter-tls" not found
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-tls\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.830847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.830777    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8c25ba8-9217-4dcc-b645-00ca89c6641e-root\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.831613 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.831011    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-textfile\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.831613 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.831140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/49e3d9c7-c939-45a4-acc1-940afde69201-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.831613 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.831197    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e3d9c7-c939-45a4-acc1-940afde69201-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.831613 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.831515    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.831613 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.831589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.832930 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.832892    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.833373 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.833345    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:41.851963 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.851936    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxtz6\" (UniqueName: \"kubernetes.io/projected/49e3d9c7-c939-45a4-acc1-940afde69201-kube-api-access-kxtz6\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:41.852076 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.852044    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z4qq\" (UniqueName: \"kubernetes.io/projected/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-kube-api-access-6z4qq\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:41.853098 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:41.853051    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qncv\" (UniqueName: \"kubernetes.io/projected/d8c25ba8-9217-4dcc-b645-00ca89c6641e-kube-api-access-2qncv\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:42.336186 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.336143    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-tls\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:42.336374 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.336193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:42.338636 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.338610    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8c25ba8-9217-4dcc-b645-00ca89c6641e-node-exporter-tls\") pod \"node-exporter-5b246\" (UID: \"d8c25ba8-9217-4dcc-b645-00ca89c6641e\") " pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:42.338742 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.338666    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49e3d9c7-c939-45a4-acc1-940afde69201-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-5qgmh\" (UID: \"49e3d9c7-c939-45a4-acc1-940afde69201\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:42.445125 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.445077    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:42.448578 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.448553    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.455148 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.455122    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:47:42.455272 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.455187    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:47:42.455272 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.455135    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:47:42.455272 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.455249    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:47:42.455769 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.455515    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:47:42.455769 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.455541    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:47:42.455769 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.455554    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:47:42.455769 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.455564    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lzdgq\""
Sep 16 18:47:42.455769 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.455638    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:47:42.456230 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.456211    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:47:42.474509 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.474488    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:47:42.480123 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.480084    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:42.482845 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.482820    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:42.519216 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.519191    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"
Sep 16 18:47:42.537636 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.537610    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.537751 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.537658    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.537805 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.537743    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.537805 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.537789    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.537877 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.537808    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-config-out\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.537877 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.537859    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-config-volume\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.537964 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.537881    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.537964 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.537908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.537964 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.537922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfx2m\" (UniqueName: \"kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-kube-api-access-cfx2m\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.538088 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.538019    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.538088 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.538056    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-web-config\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.538178 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.538098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.538178 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.538152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.541650 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.541633    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5b246"
Sep 16 18:47:42.551068 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:42.551045    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c25ba8_9217_4dcc_b645_00ca89c6641e.slice/crio-893202cf86bc6c098544d9b96749b6aac7386ea05d13199e3259abb4b93ee391 WatchSource:0}: Error finding container 893202cf86bc6c098544d9b96749b6aac7386ea05d13199e3259abb4b93ee391: Status 404 returned error can't find the container with id 893202cf86bc6c098544d9b96749b6aac7386ea05d13199e3259abb4b93ee391
Sep 16 18:47:42.610667 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.610582    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:47:42.622760 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.622730    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6619a7cc-8fc6-43ce-8a48-7a1ac0880821-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-8fwdr\" (UID: \"6619a7cc-8fc6-43ce-8a48-7a1ac0880821\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:42.639339 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-config-volume\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639339 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639338    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639486 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639364    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639587 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfx2m\" (UniqueName: \"kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-kube-api-access-cfx2m\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639653 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639713 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639650    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-web-config\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639713 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639697    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639818 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639726    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639818 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639757    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639818 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639956 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639861    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639956 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639886    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.639956 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.639913    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-config-out\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.641521 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.640180    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.641521 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.641066    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.641521 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.641431    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.642313 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.642292    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.642531 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.642511    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-config-out\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.642891 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.642858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.642966 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.642894    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.643059 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.643038    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-web-config\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.643291 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.643273    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-config-volume\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.643338 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.643277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.644224 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.644198    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.644367 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.644350    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.666075 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.666054    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfx2m\" (UniqueName: \"kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-kube-api-access-cfx2m\") pod \"alertmanager-main-0\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.678299 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.678279    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zb5dl\""
Sep 16 18:47:42.759418 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.759392    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:42.800139 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.800089    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh"]
Sep 16 18:47:42.801503 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:42.801477    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e3d9c7_c939_45a4_acc1_940afde69201.slice/crio-cfe7498fbda05c96dc3b9832d7828f7e6bc9f1d3327068495c3bd82eb78d4694 WatchSource:0}: Error finding container cfe7498fbda05c96dc3b9832d7828f7e6bc9f1d3327068495c3bd82eb78d4694: Status 404 returned error can't find the container with id cfe7498fbda05c96dc3b9832d7828f7e6bc9f1d3327068495c3bd82eb78d4694
Sep 16 18:47:42.826362 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.826338    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"
Sep 16 18:47:42.893879 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.893853    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh" event={"ID":"49e3d9c7-c939-45a4-acc1-940afde69201","Type":"ContainerStarted","Data":"cfe7498fbda05c96dc3b9832d7828f7e6bc9f1d3327068495c3bd82eb78d4694"}
Sep 16 18:47:42.895267 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:42.895242    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5b246" event={"ID":"d8c25ba8-9217-4dcc-b645-00ca89c6641e","Type":"ContainerStarted","Data":"893202cf86bc6c098544d9b96749b6aac7386ea05d13199e3259abb4b93ee391"}
Sep 16 18:47:43.059425 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.059377    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:43.062540 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:43.062504    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987de4a4_9a5a_4a49_841c_94efbff7380a.slice/crio-e1efefe03d58b6f0732964ed6770d4cfaee139fc3f485ba8bffca5d527841a7c WatchSource:0}: Error finding container e1efefe03d58b6f0732964ed6770d4cfaee139fc3f485ba8bffca5d527841a7c: Status 404 returned error can't find the container with id e1efefe03d58b6f0732964ed6770d4cfaee139fc3f485ba8bffca5d527841a7c
Sep 16 18:47:43.104418 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.104390    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr"]
Sep 16 18:47:43.108426 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:43.108353    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6619a7cc_8fc6_43ce_8a48_7a1ac0880821.slice/crio-cd1f44d749d010ee72a6360fc905436278142eb55e2a4e8c65a80dfca5071bc2 WatchSource:0}: Error finding container cd1f44d749d010ee72a6360fc905436278142eb55e2a4e8c65a80dfca5071bc2: Status 404 returned error can't find the container with id cd1f44d749d010ee72a6360fc905436278142eb55e2a4e8c65a80dfca5071bc2
Sep 16 18:47:43.143916 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.143888    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:47:43.146334 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.146313    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fa29a5-39f2-485f-b1fe-268334ec4935-metrics-certs\") pod \"network-metrics-daemon-fsv9w\" (UID: \"e7fa29a5-39f2-485f-b1fe-268334ec4935\") " pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:47:43.345637 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.345603    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtjvb\""
Sep 16 18:47:43.353479 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.353455    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fsv9w"
Sep 16 18:47:43.751941 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.751855    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fsv9w"]
Sep 16 18:47:43.905763 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.905620    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr" event={"ID":"6619a7cc-8fc6-43ce-8a48-7a1ac0880821","Type":"ContainerStarted","Data":"c68e1f3234639b10d9ea22ed453c3f6993e1fc73377ec4ab85099efc6a6286c4"}
Sep 16 18:47:43.905763 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.905694    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr" event={"ID":"6619a7cc-8fc6-43ce-8a48-7a1ac0880821","Type":"ContainerStarted","Data":"891c6ddea9fd81d1114c1d520fd8c2bd9ed3526a85372b28746d16ec7725bcb4"}
Sep 16 18:47:43.905763 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.905725    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr" event={"ID":"6619a7cc-8fc6-43ce-8a48-7a1ac0880821","Type":"ContainerStarted","Data":"cd1f44d749d010ee72a6360fc905436278142eb55e2a4e8c65a80dfca5071bc2"}
Sep 16 18:47:43.907589 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.907559    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerStarted","Data":"e1efefe03d58b6f0732964ed6770d4cfaee139fc3f485ba8bffca5d527841a7c"}
Sep 16 18:47:43.909786 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:43.909759    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5b246" event={"ID":"d8c25ba8-9217-4dcc-b645-00ca89c6641e","Type":"ContainerStarted","Data":"764284ac2292d49c00623f9469588926f0adba84f7a8d3658ecdb9ef6162beec"}
Sep 16 18:47:44.021655 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.021573    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:44.021824 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.021697    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:47:44.023330 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.023302    2567 patch_prober.go:28] interesting pod/console-c886bf65c-7xr2b container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.17:8443/health\": dial tcp 10.134.0.17:8443: connect: connection refused" start-of-body=
Sep 16 18:47:44.023466 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.023343    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-c886bf65c-7xr2b" podUID="a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" containerName="console" probeResult="failure" output="Get \"https://10.134.0.17:8443/health\": dial tcp 10.134.0.17:8443: connect: connection refused"
Sep 16 18:47:44.047494 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:44.047443    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fa29a5_39f2_485f_b1fe_268334ec4935.slice/crio-b872156890440a53907e1383ae61f4ddbbfb1fc421181c2a33e6f29f97ab45c4 WatchSource:0}: Error finding container b872156890440a53907e1383ae61f4ddbbfb1fc421181c2a33e6f29f97ab45c4: Status 404 returned error can't find the container with id b872156890440a53907e1383ae61f4ddbbfb1fc421181c2a33e6f29f97ab45c4
Sep 16 18:47:44.921182 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.921140    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr" event={"ID":"6619a7cc-8fc6-43ce-8a48-7a1ac0880821","Type":"ContainerStarted","Data":"2332cdbdf469216fac21c4f373582a27ad8ae11d826384a052ffb6713a1a7234"}
Sep 16 18:47:44.923867 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.923836    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh" event={"ID":"49e3d9c7-c939-45a4-acc1-940afde69201","Type":"ContainerStarted","Data":"2afd95517bcebe0a3506994c5469dbc6b48bab447972455b1c2540b9147202fc"}
Sep 16 18:47:44.924000 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.923897    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh" event={"ID":"49e3d9c7-c939-45a4-acc1-940afde69201","Type":"ContainerStarted","Data":"e3f47934697020fd4befd516eb89daa2d98dd5fec246ea553f36e46f84b31f29"}
Sep 16 18:47:44.924000 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.923924    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh" event={"ID":"49e3d9c7-c939-45a4-acc1-940afde69201","Type":"ContainerStarted","Data":"227057f87d207b814b24ab8f28e6c237c29c569d23f962e14d51bd1f63e0ca20"}
Sep 16 18:47:44.925600 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.925575    2567 generic.go:356] "Generic (PLEG): container finished" podID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerID="508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522" exitCode=0
Sep 16 18:47:44.925849 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.925823    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerDied","Data":"508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522"}
Sep 16 18:47:44.927490 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.927468    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fsv9w" event={"ID":"e7fa29a5-39f2-485f-b1fe-268334ec4935","Type":"ContainerStarted","Data":"b872156890440a53907e1383ae61f4ddbbfb1fc421181c2a33e6f29f97ab45c4"}
Sep 16 18:47:44.929178 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.929158    2567 generic.go:356] "Generic (PLEG): container finished" podID="d8c25ba8-9217-4dcc-b645-00ca89c6641e" containerID="764284ac2292d49c00623f9469588926f0adba84f7a8d3658ecdb9ef6162beec" exitCode=0
Sep 16 18:47:44.929248 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.929230    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5b246" event={"ID":"d8c25ba8-9217-4dcc-b645-00ca89c6641e","Type":"ContainerDied","Data":"764284ac2292d49c00623f9469588926f0adba84f7a8d3658ecdb9ef6162beec"}
Sep 16 18:47:44.946587 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.946537    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8fwdr" podStartSLOduration=2.638264205 podStartE2EDuration="3.946498074s" totalImagesPullingTime="1.308233869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:44.945341259 +0000 UTC m=+132.133787941" watchObservedRunningTime="2026-09-16 18:47:44.946498074 +0000 UTC m=+132.134944757"
Sep 16 18:47:44.971472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:44.971423    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5qgmh" podStartSLOduration=2.23120887 podStartE2EDuration="3.971403305s" totalImagesPullingTime="1.740194435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:44.970303179 +0000 UTC m=+132.158749862" watchObservedRunningTime="2026-09-16 18:47:44.971403305 +0000 UTC m=+132.159849988"
Sep 16 18:47:45.936123 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:45.936077    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fsv9w" event={"ID":"e7fa29a5-39f2-485f-b1fe-268334ec4935","Type":"ContainerStarted","Data":"668c1562545a4d2b05adf16e3692dc1ae1355c0af4d3e7cf26d82ffdd450d999"}
Sep 16 18:47:45.936557 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:45.936169    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fsv9w" event={"ID":"e7fa29a5-39f2-485f-b1fe-268334ec4935","Type":"ContainerStarted","Data":"fd5a5673aaf335d109dd4741daf5acb36806e9466f9f5c70a217ff2af565e4bd"}
Sep 16 18:47:45.938755 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:45.938724    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5b246" event={"ID":"d8c25ba8-9217-4dcc-b645-00ca89c6641e","Type":"ContainerStarted","Data":"025f537a4e47abc160d85e2949df9a017c7d5da67e5509c0c6878604d7fdbdaf"}
Sep 16 18:47:45.938881 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:45.938788    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5b246" event={"ID":"d8c25ba8-9217-4dcc-b645-00ca89c6641e","Type":"ContainerStarted","Data":"d6e0a457be38387bf449437804e9167ce1ca351b4049fe8537d94e80d37d2518"}
Sep 16 18:47:45.962236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:45.962179    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fsv9w" podStartSLOduration=131.756305186 podStartE2EDuration="2m12.962159779s" totalImagesPullingTime="1.205854593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:45.959549266 +0000 UTC m=+133.147995961" watchObservedRunningTime="2026-09-16 18:47:45.962159779 +0000 UTC m=+133.150606460"
Sep 16 18:47:45.995221 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:45.995165    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5b246" podStartSLOduration=3.9806650550000002 podStartE2EDuration="4.995149595s" totalImagesPullingTime="942.040477ms" totalInitContainerRuntime="72.444063ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:45.993904717 +0000 UTC m=+133.182351428" watchObservedRunningTime="2026-09-16 18:47:45.995149595 +0000 UTC m=+133.183596278"
Sep 16 18:47:46.149177 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.149132    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:46.149354 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.149202    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:47:46.150703 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.150674    2567 patch_prober.go:28] interesting pod/console-65799dbbd-v4tfv container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused" start-of-body=
Sep 16 18:47:46.150841 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.150721    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-65799dbbd-v4tfv" podUID="0a8d8757-4193-4971-a87a-2dc7b8ba30fe" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused"
Sep 16 18:47:46.261743 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.261658    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c886bf65c-7xr2b"]
Sep 16 18:47:46.273181 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.273150    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-56b5948bdb-qxwmw"]
Sep 16 18:47:46.277973 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.277954    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.282049 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.281949    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:47:46.282801 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.282713    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-efvb8vqnda7to\""
Sep 16 18:47:46.282801 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.282732    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-z656r\""
Sep 16 18:47:46.282801 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.282732    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:47:46.290675 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.290655    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:47:46.291801 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.291780    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:47:46.296857 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.296836    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56b5948bdb-qxwmw"]
Sep 16 18:47:46.370324 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.370279    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f967777b9-5h7rt"]
Sep 16 18:47:46.374445 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.374400    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f967777b9-5h7rt"]
Sep 16 18:47:46.374582 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.374503    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.377461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.377436    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwln4\" (UniqueName: \"kubernetes.io/projected/541e723c-a0a6-447f-b16f-72359bc7c382-kube-api-access-kwln4\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.377598 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.377484    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/541e723c-a0a6-447f-b16f-72359bc7c382-audit-log\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.377666 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.377593    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/541e723c-a0a6-447f-b16f-72359bc7c382-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.377666 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.377632    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/541e723c-a0a6-447f-b16f-72359bc7c382-secret-metrics-server-tls\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.377767 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.377664    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/541e723c-a0a6-447f-b16f-72359bc7c382-secret-metrics-server-client-certs\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.377767 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.377694    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/541e723c-a0a6-447f-b16f-72359bc7c382-client-ca-bundle\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.377767 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.377751    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/541e723c-a0a6-447f-b16f-72359bc7c382-metrics-server-audit-profiles\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.478937 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.478896    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-oauth-config\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.479191 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.478943    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-serving-cert\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.479191 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479003    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/541e723c-a0a6-447f-b16f-72359bc7c382-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.479191 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/541e723c-a0a6-447f-b16f-72359bc7c382-secret-metrics-server-tls\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.479191 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479059    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/541e723c-a0a6-447f-b16f-72359bc7c382-secret-metrics-server-client-certs\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.479191 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479079    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/541e723c-a0a6-447f-b16f-72359bc7c382-client-ca-bundle\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.479191 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-trusted-ca-bundle\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.479191 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479140    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-oauth-serving-cert\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.479191 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/541e723c-a0a6-447f-b16f-72359bc7c382-metrics-server-audit-profiles\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.479191 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479192    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-service-ca\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.479682 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479252    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-config\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.479682 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwln4\" (UniqueName: \"kubernetes.io/projected/541e723c-a0a6-447f-b16f-72359bc7c382-kube-api-access-kwln4\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.479682 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479300    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ttrq\" (UniqueName: \"kubernetes.io/projected/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-kube-api-access-2ttrq\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.479682 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479324    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/541e723c-a0a6-447f-b16f-72359bc7c382-audit-log\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.479682 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479672    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/541e723c-a0a6-447f-b16f-72359bc7c382-audit-log\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.479925 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.479795    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/541e723c-a0a6-447f-b16f-72359bc7c382-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.480164 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.480144    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/541e723c-a0a6-447f-b16f-72359bc7c382-metrics-server-audit-profiles\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.481698 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.481667    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/541e723c-a0a6-447f-b16f-72359bc7c382-secret-metrics-server-tls\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.481783 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.481704    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/541e723c-a0a6-447f-b16f-72359bc7c382-client-ca-bundle\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.481783 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.481716    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/541e723c-a0a6-447f-b16f-72359bc7c382-secret-metrics-server-client-certs\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.492052 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.492032    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwln4\" (UniqueName: \"kubernetes.io/projected/541e723c-a0a6-447f-b16f-72359bc7c382-kube-api-access-kwln4\") pod \"metrics-server-56b5948bdb-qxwmw\" (UID: \"541e723c-a0a6-447f-b16f-72359bc7c382\") " pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.580628 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.580606    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-oauth-config\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.580727 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.580638    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-serving-cert\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.580727 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.580671    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-trusted-ca-bundle\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.580727 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.580689    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-oauth-serving-cert\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.580727 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.580714    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-service-ca\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.580979 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.580853    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-config\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.580979 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.580910    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ttrq\" (UniqueName: \"kubernetes.io/projected/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-kube-api-access-2ttrq\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.581504 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.581479    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-oauth-serving-cert\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.581612 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.581556    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-service-ca\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.581612 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.581568    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-config\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.581740 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.581630    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-trusted-ca-bundle\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.582966 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.582947    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-serving-cert\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.583041 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.582979    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-oauth-config\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.590264 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.590243    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:47:46.594137 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.594097    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ttrq\" (UniqueName: \"kubernetes.io/projected/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-kube-api-access-2ttrq\") pod \"console-6f967777b9-5h7rt\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") " pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.686163 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.686138    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:46.888569 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.888544    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56b5948bdb-qxwmw"]
Sep 16 18:47:46.890315 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:46.890291    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541e723c_a0a6_447f_b16f_72359bc7c382.slice/crio-c6cf631ce58404ae10e2e4baf01f46c57cca1524a108a65b2712458805bea575 WatchSource:0}: Error finding container c6cf631ce58404ae10e2e4baf01f46c57cca1524a108a65b2712458805bea575: Status 404 returned error can't find the container with id c6cf631ce58404ae10e2e4baf01f46c57cca1524a108a65b2712458805bea575
Sep 16 18:47:46.943569 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.943539    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw" event={"ID":"541e723c-a0a6-447f-b16f-72359bc7c382","Type":"ContainerStarted","Data":"c6cf631ce58404ae10e2e4baf01f46c57cca1524a108a65b2712458805bea575"}
Sep 16 18:47:46.946500 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.946480    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerStarted","Data":"eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e"}
Sep 16 18:47:46.946620 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.946547    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerStarted","Data":"be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b"}
Sep 16 18:47:46.946620 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.946591    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerStarted","Data":"eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343"}
Sep 16 18:47:46.946726 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.946626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerStarted","Data":"c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2"}
Sep 16 18:47:46.946726 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.946672    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerStarted","Data":"ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de"}
Sep 16 18:47:46.978175 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:46.978150    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f967777b9-5h7rt"]
Sep 16 18:47:46.980382 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:46.980360    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a51d2e1_0496_4ea6_9f30_9c4a7ecf1d91.slice/crio-a02e124e796afabd4267e83a96f26a1651d00bbb813be886538af9e93a736fa0 WatchSource:0}: Error finding container a02e124e796afabd4267e83a96f26a1651d00bbb813be886538af9e93a736fa0: Status 404 returned error can't find the container with id a02e124e796afabd4267e83a96f26a1651d00bbb813be886538af9e93a736fa0
Sep 16 18:47:47.909955 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.907641    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:47.911468 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.911406    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.915616 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.915590    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:47:47.915895 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.915878    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:47:47.916168 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.916153    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:47:47.916231 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.916186    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:47:47.926589 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.926563    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:47:47.926890 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.926864    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:47:47.926890 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.926882    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:47:47.927936 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.927914    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:47:47.928047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.927944    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8bj3qtqeotiuf\""
Sep 16 18:47:47.928047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.927959    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n5hnw\""
Sep 16 18:47:47.928047 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.927915    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:47:47.928198 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.927914    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:47:47.928393 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.928377    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:47:47.929472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.929453    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:47:47.931061 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.931044    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:47:47.936035 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.936007    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:47:47.953336 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.953305    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f967777b9-5h7rt" event={"ID":"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91","Type":"ContainerStarted","Data":"d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0"}
Sep 16 18:47:47.953721 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.953359    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f967777b9-5h7rt" event={"ID":"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91","Type":"ContainerStarted","Data":"a02e124e796afabd4267e83a96f26a1651d00bbb813be886538af9e93a736fa0"}
Sep 16 18:47:47.957276 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.957256    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerStarted","Data":"15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486"}
Sep 16 18:47:47.963164 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.963139    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:47.998666 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.998635    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.998789 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.998670    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.998789 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.998703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.998789 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.998727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.998922 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.998826    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.998922 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.998860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.998922 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.998883    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.998922 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.998897    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999193 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.998922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999193 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.998966    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999193 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.999012    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999193 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.999043    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999193 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.999082    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999193 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.999154    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999453 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.999209    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999453 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.999231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2pff\" (UniqueName: \"kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-kube-api-access-w2pff\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999453 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.999258    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999453 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.999291    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999453 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.999316    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.999453 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:47.999387    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100245 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100245 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100200    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100245 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100222    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100520 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100249    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100520 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100520 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100348    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100520 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100403    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100520 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100467    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100583    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2pff\" (UniqueName: \"kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-kube-api-access-w2pff\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.100983 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.100775    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.101244 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.101218    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.101363 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.101256    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.101363 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.101312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.101363 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.101341    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.101511 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.101423    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.101511 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.101477    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.101870 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.101835    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.102083 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.102061    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.102606 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.102324    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.102606 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.102341    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.102606 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.102389    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.103874 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.103549    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.104837 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.104401    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.104837 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.104800    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.104994 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.104875    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.105243 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.105220    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.105452 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.105428    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.106871 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.106494    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.106871 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.106835    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.107663 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.107620    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.107793 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.107747    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.107901 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.107864    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.108073 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.108051    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.108479 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.108456    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.108561 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.108522    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.119562 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.119504    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.110333561 podStartE2EDuration="6.119484575s" totalImagesPullingTime="3.983134643s" totalInitContainerRuntime="26.016371ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:48.118538046 +0000 UTC m=+135.306984742" watchObservedRunningTime="2026-09-16 18:47:48.119484575 +0000 UTC m=+135.307931262"
Sep 16 18:47:48.133512 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.133488    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2pff\" (UniqueName: \"kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-kube-api-access-w2pff\") pod \"prometheus-k8s-0\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.223255 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.223206    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:48.557791 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.557740    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f967777b9-5h7rt" podStartSLOduration=2.5577224689999998 podStartE2EDuration="2.557722469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:48.18031652 +0000 UTC m=+135.368763203" watchObservedRunningTime="2026-09-16 18:47:48.557722469 +0000 UTC m=+135.746169153"
Sep 16 18:47:48.558819 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.558799    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:48.561337 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:47:48.561308    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff00e73_8b62_4397_bb86_4f6fba1a9842.slice/crio-6b2f31008da936448755a1987ecabbc9bd3de4974dec6f207cae76d972846ba9 WatchSource:0}: Error finding container 6b2f31008da936448755a1987ecabbc9bd3de4974dec6f207cae76d972846ba9: Status 404 returned error can't find the container with id 6b2f31008da936448755a1987ecabbc9bd3de4974dec6f207cae76d972846ba9
Sep 16 18:47:48.965745 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.965710    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw" event={"ID":"541e723c-a0a6-447f-b16f-72359bc7c382","Type":"ContainerStarted","Data":"213d25c5ad5e2296283345ca30945999ba326c4e5eadd5ce58715e9b3ac27fb3"}
Sep 16 18:47:48.967375 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.967347    2567 generic.go:356] "Generic (PLEG): container finished" podID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerID="bdc92997f2c8980c94eb45352781347e31c82e609e2e20e6512441b8cc4e6a1b" exitCode=0
Sep 16 18:47:48.967476 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.967425    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerDied","Data":"bdc92997f2c8980c94eb45352781347e31c82e609e2e20e6512441b8cc4e6a1b"}
Sep 16 18:47:48.967531 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:48.967478    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerStarted","Data":"6b2f31008da936448755a1987ecabbc9bd3de4974dec6f207cae76d972846ba9"}
Sep 16 18:47:49.005532 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:49.005486    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw" podStartSLOduration=1.606677728 podStartE2EDuration="3.005471356s" totalImagesPullingTime="1.398793628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:49.005308772 +0000 UTC m=+136.193755467" watchObservedRunningTime="2026-09-16 18:47:49.005471356 +0000 UTC m=+136.193918038"
Sep 16 18:47:49.242206 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:49.242130    2567 patch_prober.go:28] interesting pod/image-registry-56dc97c7df-vcth6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:47:49.242206 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:49.242184    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6" podUID="4d974f52-86bb-420f-a91d-39f2dfa2c62b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:47:51.985729 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:51.985701    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerStarted","Data":"70dbd150ed3460c4b22fb06d4549adb4a0e82a3c74928efe8c6743a8b086e57d"}
Sep 16 18:47:52.994050 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:52.994015    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerStarted","Data":"355812060b7b89f38af2a71b0e1041bb795e9794a97e996ecac87cd62a61070d"}
Sep 16 18:47:54.001811 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.001724    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerStarted","Data":"bceee3375960bc42b65525334f59c846bdf5f106946e70a1fb5512323b6fe71e"}
Sep 16 18:47:54.001811 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.001793    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerStarted","Data":"71edf8732ceccf35227a0bc5729fb7792eab04c64fafdd4dc96fb4b0701f61b0"}
Sep 16 18:47:54.002510 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.001832    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerStarted","Data":"276f062aff5f5007c31890c992af0023e10b5235da1d0ae6ad636dba29b290e0"}
Sep 16 18:47:54.002510 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.001865    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerStarted","Data":"cef259f8f623d3166d377d23dcef060ee65ba2aaed7cc57c8dfdeeb19593c351"}
Sep 16 18:47:54.057101 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.057055    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.478044716 podStartE2EDuration="7.057041234s" totalImagesPullingTime="4.557011853s" totalInitContainerRuntime="21.984665ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:54.056064223 +0000 UTC m=+141.244510928" watchObservedRunningTime="2026-09-16 18:47:54.057041234 +0000 UTC m=+141.245487915"
Sep 16 18:47:54.255544 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.255465    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6" podUID="4d974f52-86bb-420f-a91d-39f2dfa2c62b" containerName="registry" containerID="cri-o://84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c" gracePeriod=30
Sep 16 18:47:54.494466 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.494444    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:54.672264 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.672223    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls\") pod \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") "
Sep 16 18:47:54.672264 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.672264    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-trusted-ca\") pod \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") "
Sep 16 18:47:54.672477 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.672295    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d974f52-86bb-420f-a91d-39f2dfa2c62b-ca-trust-extracted\") pod \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") "
Sep 16 18:47:54.672477 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.672319    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-image-registry-private-configuration\") pod \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") "
Sep 16 18:47:54.672477 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.672361    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-bound-sa-token\") pod \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") "
Sep 16 18:47:54.672477 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.672450    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-installation-pull-secrets\") pod \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") "
Sep 16 18:47:54.672645 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.672501    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp8mv\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-kube-api-access-fp8mv\") pod \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") "
Sep 16 18:47:54.672645 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.672530    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-certificates\") pod \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\" (UID: \"4d974f52-86bb-420f-a91d-39f2dfa2c62b\") "
Sep 16 18:47:54.673194 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.673159    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4d974f52-86bb-420f-a91d-39f2dfa2c62b" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:54.673300 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.673200    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4d974f52-86bb-420f-a91d-39f2dfa2c62b" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:54.675219 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.675192    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-kube-api-access-fp8mv" (OuterVolumeSpecName: "kube-api-access-fp8mv") pod "4d974f52-86bb-420f-a91d-39f2dfa2c62b" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b"). InnerVolumeSpecName "kube-api-access-fp8mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:54.675311 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.675233    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4d974f52-86bb-420f-a91d-39f2dfa2c62b" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:54.675311 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.675287    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4d974f52-86bb-420f-a91d-39f2dfa2c62b" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:54.675463 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.675441    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4d974f52-86bb-420f-a91d-39f2dfa2c62b" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:54.675463 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.675448    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4d974f52-86bb-420f-a91d-39f2dfa2c62b" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:54.680683 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.680660    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d974f52-86bb-420f-a91d-39f2dfa2c62b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4d974f52-86bb-420f-a91d-39f2dfa2c62b" (UID: "4d974f52-86bb-420f-a91d-39f2dfa2c62b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:54.774099 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.774076    2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-bound-sa-token\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:47:54.774099 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.774097    2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-installation-pull-secrets\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:47:54.774272 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.774106    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp8mv\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-kube-api-access-fp8mv\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:47:54.774272 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.774139    2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-certificates\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:47:54.774272 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.774147    2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d974f52-86bb-420f-a91d-39f2dfa2c62b-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:47:54.774272 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.774155    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d974f52-86bb-420f-a91d-39f2dfa2c62b-trusted-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:47:54.774272 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.774163    2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d974f52-86bb-420f-a91d-39f2dfa2c62b-ca-trust-extracted\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:47:54.774272 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:54.774172    2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d974f52-86bb-420f-a91d-39f2dfa2c62b-image-registry-private-configuration\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:47:55.007456 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:55.007371    2567 generic.go:356] "Generic (PLEG): container finished" podID="4d974f52-86bb-420f-a91d-39f2dfa2c62b" containerID="84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c" exitCode=0
Sep 16 18:47:55.007456 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:55.007424    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6"
Sep 16 18:47:55.007936 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:55.007460    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6" event={"ID":"4d974f52-86bb-420f-a91d-39f2dfa2c62b","Type":"ContainerDied","Data":"84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c"}
Sep 16 18:47:55.007936 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:55.007511    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56dc97c7df-vcth6" event={"ID":"4d974f52-86bb-420f-a91d-39f2dfa2c62b","Type":"ContainerDied","Data":"1057faf161c12c8541f07377a5f94a53ec299f7d1f54097e3c6f081d0e31996e"}
Sep 16 18:47:55.007936 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:55.007532    2567 scope.go:117] "RemoveContainer" containerID="84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c"
Sep 16 18:47:55.016157 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:55.016141    2567 scope.go:117] "RemoveContainer" containerID="84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c"
Sep 16 18:47:55.016398 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:47:55.016377    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c\": container with ID starting with 84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c not found: ID does not exist" containerID="84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c"
Sep 16 18:47:55.016439 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:55.016406    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c"} err="failed to get container status \"84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c\": rpc error: code = NotFound desc = could not find container \"84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c\": container with ID starting with 84f96560f9a8702be5788da06ce46eb9a061bb0673d514e8d8b77aa7f0f7004c not found: ID does not exist"
Sep 16 18:47:55.035833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:55.035813    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56dc97c7df-vcth6"]
Sep 16 18:47:55.045805 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:55.045783    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56dc97c7df-vcth6"]
Sep 16 18:47:55.333571 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:55.333480    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d974f52-86bb-420f-a91d-39f2dfa2c62b" path="/var/lib/kubelet/pods/4d974f52-86bb-420f-a91d-39f2dfa2c62b/volumes"
Sep 16 18:47:56.149290 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:56.149254    2567 patch_prober.go:28] interesting pod/console-65799dbbd-v4tfv container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused" start-of-body=
Sep 16 18:47:56.149675 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:56.149306    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-65799dbbd-v4tfv" podUID="0a8d8757-4193-4971-a87a-2dc7b8ba30fe" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused"
Sep 16 18:47:56.687178 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:56.687142    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:56.687347 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:56.687211    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:47:56.688445 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:56.688424    2567 patch_prober.go:28] interesting pod/console-6f967777b9-5h7rt container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused" start-of-body=
Sep 16 18:47:56.688526 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:56.688471    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6f967777b9-5h7rt" podUID="8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" containerName="console" probeResult="failure" output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused"
Sep 16 18:47:58.223365 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:47:58.223329    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:06.149241 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:06.149204    2567 patch_prober.go:28] interesting pod/console-65799dbbd-v4tfv container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused" start-of-body=
Sep 16 18:48:06.149614 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:06.149259    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-65799dbbd-v4tfv" podUID="0a8d8757-4193-4971-a87a-2dc7b8ba30fe" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused"
Sep 16 18:48:06.591346 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:06.591256    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:48:06.591346 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:06.591335    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:48:06.686572 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:06.686543    2567 patch_prober.go:28] interesting pod/console-6f967777b9-5h7rt container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused" start-of-body=
Sep 16 18:48:06.686714 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:06.686594    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6f967777b9-5h7rt" podUID="8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" containerName="console" probeResult="failure" output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused"
Sep 16 18:48:10.122075 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:48:10.122032    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mf48l" podUID="3b545178-c81e-4173-9675-50aaa359f497"
Sep 16 18:48:11.071669 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.071630    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mf48l"
Sep 16 18:48:11.285259 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.285202    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c886bf65c-7xr2b" podUID="a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" containerName="console" containerID="cri-o://019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632" gracePeriod=15
Sep 16 18:48:11.531701 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.531678    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:48:11.628315 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.628288    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-oauth-config\") pod \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") "
Sep 16 18:48:11.628472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.628337    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-oauth-serving-cert\") pod \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") "
Sep 16 18:48:11.628472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.628361    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-config\") pod \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") "
Sep 16 18:48:11.628472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.628384    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-service-ca\") pod \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") "
Sep 16 18:48:11.628472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.628423    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-serving-cert\") pod \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") "
Sep 16 18:48:11.628472 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.628461    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsmfs\" (UniqueName: \"kubernetes.io/projected/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-kube-api-access-fsmfs\") pod \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\" (UID: \"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8\") "
Sep 16 18:48:11.628781 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.628750    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" (UID: "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:11.628781 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.628774    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-config" (OuterVolumeSpecName: "console-config") pod "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" (UID: "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:11.628904 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.628807    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-service-ca" (OuterVolumeSpecName: "service-ca") pod "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" (UID: "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:11.630534 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.630507    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-kube-api-access-fsmfs" (OuterVolumeSpecName: "kube-api-access-fsmfs") pod "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" (UID: "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8"). InnerVolumeSpecName "kube-api-access-fsmfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:11.630661 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.630637    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" (UID: "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:11.630661 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.630647    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" (UID: "a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:11.729255 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.729196    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-oauth-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:11.729255 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.729219    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-oauth-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:11.729255 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.729228    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:11.729255 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.729236    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:11.729255 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.729244    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-console-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:11.729255 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:11.729252    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsmfs\" (UniqueName: \"kubernetes.io/projected/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8-kube-api-access-fsmfs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:12.076371 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:12.076294    2567 generic.go:356] "Generic (PLEG): container finished" podID="a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" containerID="019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632" exitCode=0
Sep 16 18:48:12.076371 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:12.076348    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c886bf65c-7xr2b"
Sep 16 18:48:12.076527 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:12.076352    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c886bf65c-7xr2b" event={"ID":"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8","Type":"ContainerDied","Data":"019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632"}
Sep 16 18:48:12.076527 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:12.076483    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c886bf65c-7xr2b" event={"ID":"a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8","Type":"ContainerDied","Data":"d878857228b9609b8bc067dfbd49fd6519a8c9fe6885713f1cbcce5059af355f"}
Sep 16 18:48:12.076527 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:12.076510    2567 scope.go:117] "RemoveContainer" containerID="019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632"
Sep 16 18:48:12.084824 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:12.084807    2567 scope.go:117] "RemoveContainer" containerID="019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632"
Sep 16 18:48:12.085099 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:48:12.085082    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632\": container with ID starting with 019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632 not found: ID does not exist" containerID="019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632"
Sep 16 18:48:12.085210 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:12.085128    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632"} err="failed to get container status \"019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632\": rpc error: code = NotFound desc = could not find container \"019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632\": container with ID starting with 019ad452ce1da87ddcf178644227af00f620d3439ad904dbf838f8dfddcd4632 not found: ID does not exist"
Sep 16 18:48:12.099667 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:12.099635    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c886bf65c-7xr2b"]
Sep 16 18:48:12.103713 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:12.103693    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c886bf65c-7xr2b"]
Sep 16 18:48:13.332890 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:13.332860    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" path="/var/lib/kubelet/pods/a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8/volumes"
Sep 16 18:48:14.961907 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:14.961871    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:48:14.964131 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:14.964099    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b545178-c81e-4173-9675-50aaa359f497-metrics-tls\") pod \"dns-default-mf48l\" (UID: \"3b545178-c81e-4173-9675-50aaa359f497\") " pod="openshift-dns/dns-default-mf48l"
Sep 16 18:48:14.976506 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:14.976479    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k8559\""
Sep 16 18:48:14.983525 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:14.983507    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mf48l"
Sep 16 18:48:15.273584 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:15.273558    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mf48l"]
Sep 16 18:48:15.276318 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:48:15.276290    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b545178_c81e_4173_9675_50aaa359f497.slice/crio-7161241d7dde5a385fe8d55e7212cc24e912c2a84dae8de7e4e523403751e9cc WatchSource:0}: Error finding container 7161241d7dde5a385fe8d55e7212cc24e912c2a84dae8de7e4e523403751e9cc: Status 404 returned error can't find the container with id 7161241d7dde5a385fe8d55e7212cc24e912c2a84dae8de7e4e523403751e9cc
Sep 16 18:48:16.095605 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:16.095569    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mf48l" event={"ID":"3b545178-c81e-4173-9675-50aaa359f497","Type":"ContainerStarted","Data":"7161241d7dde5a385fe8d55e7212cc24e912c2a84dae8de7e4e523403751e9cc"}
Sep 16 18:48:16.154009 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:16.153973    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:48:16.157959 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:16.157935    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:48:16.690896 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:16.690866    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:48:16.694943 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:16.694915    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:48:16.798715 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:16.798691    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65799dbbd-v4tfv"]
Sep 16 18:48:17.102709 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:17.102619    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mf48l" event={"ID":"3b545178-c81e-4173-9675-50aaa359f497","Type":"ContainerStarted","Data":"5aaa86c7c48341aa31e1b0547bd0e4fe0490607b9d31e8bcb50cc388e1737f7c"}
Sep 16 18:48:17.102709 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:17.102686    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mf48l" event={"ID":"3b545178-c81e-4173-9675-50aaa359f497","Type":"ContainerStarted","Data":"e8279d3698cfa41d2b0d24ebdab714bc3efcd9c10a5a7152b4373e7acc4632fd"}
Sep 16 18:48:17.103199 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:17.102747    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mf48l"
Sep 16 18:48:17.130368 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:17.130320    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mf48l" podStartSLOduration=128.712748549 podStartE2EDuration="2m10.130305927s" totalImagesPullingTime="1.417557378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:17.129453714 +0000 UTC m=+164.317900395" watchObservedRunningTime="2026-09-16 18:48:17.130305927 +0000 UTC m=+164.318752609"
Sep 16 18:48:26.596465 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:26.596381    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:48:26.600180 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:26.600157    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-56b5948bdb-qxwmw"
Sep 16 18:48:27.109451 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:27.109423    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mf48l"
Sep 16 18:48:32.162546 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:32.162511    2567 generic.go:356] "Generic (PLEG): container finished" podID="81bd4b21-6b45-4504-b47f-a6201451b987" containerID="0cf06ef482fa453ff11ad734896a2491875a30d62d129848ae1ebe7c70664f46" exitCode=0
Sep 16 18:48:32.162911 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:32.162582    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-2jhcw" event={"ID":"81bd4b21-6b45-4504-b47f-a6201451b987","Type":"ContainerDied","Data":"0cf06ef482fa453ff11ad734896a2491875a30d62d129848ae1ebe7c70664f46"}
Sep 16 18:48:32.162978 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:32.162962    2567 scope.go:117] "RemoveContainer" containerID="0cf06ef482fa453ff11ad734896a2491875a30d62d129848ae1ebe7c70664f46"
Sep 16 18:48:33.171026 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:33.170992    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-2jhcw" event={"ID":"81bd4b21-6b45-4504-b47f-a6201451b987","Type":"ContainerStarted","Data":"427a83baf825d410bc222242d47c046001375b63dd2c3789c331b9047db88328"}
Sep 16 18:48:43.125070 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.125002    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65799dbbd-v4tfv" podUID="0a8d8757-4193-4971-a87a-2dc7b8ba30fe" containerName="console" containerID="cri-o://8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3" gracePeriod=15
Sep 16 18:48:43.382148 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.382065    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:48:43.431084 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431047    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-trusted-ca-bundle\") pod \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") "
Sep 16 18:48:43.431268 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431094    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-serving-cert\") pod \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") "
Sep 16 18:48:43.431268 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431159    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-config\") pod \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") "
Sep 16 18:48:43.431404 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431361    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-oauth-serving-cert\") pod \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") "
Sep 16 18:48:43.431460 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431423    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvvhk\" (UniqueName: \"kubernetes.io/projected/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-kube-api-access-nvvhk\") pod \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") "
Sep 16 18:48:43.431507 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431481    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0a8d8757-4193-4971-a87a-2dc7b8ba30fe" (UID: "0a8d8757-4193-4971-a87a-2dc7b8ba30fe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:43.431563 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431498    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-oauth-config\") pod \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") "
Sep 16 18:48:43.431563 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431522    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-config" (OuterVolumeSpecName: "console-config") pod "0a8d8757-4193-4971-a87a-2dc7b8ba30fe" (UID: "0a8d8757-4193-4971-a87a-2dc7b8ba30fe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:43.431659 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431624    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-service-ca\") pod \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\" (UID: \"0a8d8757-4193-4971-a87a-2dc7b8ba30fe\") "
Sep 16 18:48:43.431894 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431874    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-trusted-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:43.431991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431903    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:43.431991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.431868    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0a8d8757-4193-4971-a87a-2dc7b8ba30fe" (UID: "0a8d8757-4193-4971-a87a-2dc7b8ba30fe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:43.432141 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.432020    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-service-ca" (OuterVolumeSpecName: "service-ca") pod "0a8d8757-4193-4971-a87a-2dc7b8ba30fe" (UID: "0a8d8757-4193-4971-a87a-2dc7b8ba30fe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:43.433657 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.433632    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0a8d8757-4193-4971-a87a-2dc7b8ba30fe" (UID: "0a8d8757-4193-4971-a87a-2dc7b8ba30fe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:43.433657 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.433639    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-kube-api-access-nvvhk" (OuterVolumeSpecName: "kube-api-access-nvvhk") pod "0a8d8757-4193-4971-a87a-2dc7b8ba30fe" (UID: "0a8d8757-4193-4971-a87a-2dc7b8ba30fe"). InnerVolumeSpecName "kube-api-access-nvvhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:43.433833 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.433818    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0a8d8757-4193-4971-a87a-2dc7b8ba30fe" (UID: "0a8d8757-4193-4971-a87a-2dc7b8ba30fe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:43.533406 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.533359    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-oauth-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:43.533406 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.533406    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:43.533602 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.533420    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-console-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:43.533602 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.533432    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-oauth-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:43.533602 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:43.533445    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvhk\" (UniqueName: \"kubernetes.io/projected/0a8d8757-4193-4971-a87a-2dc7b8ba30fe-kube-api-access-nvvhk\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:48:44.218799 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:44.218761    2567 generic.go:356] "Generic (PLEG): container finished" podID="0a8d8757-4193-4971-a87a-2dc7b8ba30fe" containerID="8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3" exitCode=0
Sep 16 18:48:44.218799 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:44.218800    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65799dbbd-v4tfv"
Sep 16 18:48:44.219293 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:44.218833    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65799dbbd-v4tfv" event={"ID":"0a8d8757-4193-4971-a87a-2dc7b8ba30fe","Type":"ContainerDied","Data":"8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3"}
Sep 16 18:48:44.219293 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:44.218921    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65799dbbd-v4tfv" event={"ID":"0a8d8757-4193-4971-a87a-2dc7b8ba30fe","Type":"ContainerDied","Data":"db8c60a45ab97443d9bc1bb5e333ddc29ddcf4a96c2a2542b8071f4b35f13b0e"}
Sep 16 18:48:44.219293 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:44.218955    2567 scope.go:117] "RemoveContainer" containerID="8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3"
Sep 16 18:48:44.228151 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:44.228098    2567 scope.go:117] "RemoveContainer" containerID="8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3"
Sep 16 18:48:44.228438 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:48:44.228420    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3\": container with ID starting with 8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3 not found: ID does not exist" containerID="8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3"
Sep 16 18:48:44.228498 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:44.228464    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3"} err="failed to get container status \"8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3\": rpc error: code = NotFound desc = could not find container \"8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3\": container with ID starting with 8bc66088a180650366ccd8e0a6d16f926be2e74346df4834ea1d1ef6c4fb7ed3 not found: ID does not exist"
Sep 16 18:48:44.241532 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:44.241505    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65799dbbd-v4tfv"]
Sep 16 18:48:44.245700 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:44.245677    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65799dbbd-v4tfv"]
Sep 16 18:48:45.333491 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:45.333450    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a8d8757-4193-4971-a87a-2dc7b8ba30fe" path="/var/lib/kubelet/pods/0a8d8757-4193-4971-a87a-2dc7b8ba30fe/volumes"
Sep 16 18:48:48.223886 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:48.223849    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:48.243693 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:48.243660    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:48.260397 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:48:48.260371    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:01.822737 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:01.822700    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:01.823355 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:01.823092    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="alertmanager" containerID="cri-o://ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de" gracePeriod=120
Sep 16 18:49:01.823355 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:01.823160    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy-metric" containerID="cri-o://eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e" gracePeriod=120
Sep 16 18:49:01.823355 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:01.823184    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="prom-label-proxy" containerID="cri-o://15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486" gracePeriod=120
Sep 16 18:49:01.823355 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:01.823216    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy-web" containerID="cri-o://eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343" gracePeriod=120
Sep 16 18:49:01.823355 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:01.823234    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="config-reloader" containerID="cri-o://c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2" gracePeriod=120
Sep 16 18:49:01.823355 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:01.823225    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy" containerID="cri-o://be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b" gracePeriod=120
Sep 16 18:49:02.305483 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:02.305452    2567 generic.go:356] "Generic (PLEG): container finished" podID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerID="15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486" exitCode=0
Sep 16 18:49:02.305483 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:02.305477    2567 generic.go:356] "Generic (PLEG): container finished" podID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerID="be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b" exitCode=0
Sep 16 18:49:02.305483 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:02.305486    2567 generic.go:356] "Generic (PLEG): container finished" podID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerID="c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2" exitCode=0
Sep 16 18:49:02.305483 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:02.305494    2567 generic.go:356] "Generic (PLEG): container finished" podID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerID="ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de" exitCode=0
Sep 16 18:49:02.305756 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:02.305528    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerDied","Data":"15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486"}
Sep 16 18:49:02.305756 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:02.305591    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerDied","Data":"be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b"}
Sep 16 18:49:02.305756 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:02.305619    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerDied","Data":"c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2"}
Sep 16 18:49:02.305756 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:02.305643    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerDied","Data":"ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de"}
Sep 16 18:49:03.079816 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.079786    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.198730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198647    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-metrics-client-ca\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.198730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198719    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.198952 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198744    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-cluster-tls-config\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.198952 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198774    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-config-out\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.198952 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198807    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-trusted-ca-bundle\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.198952 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198838    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfx2m\" (UniqueName: \"kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-kube-api-access-cfx2m\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.198952 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198865    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-tls-assets\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.198952 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198892    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-web-config\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.198952 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198924    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-web\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.199406 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198960    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-config-volume\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.199406 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.198993    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-main-db\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.199406 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.199022    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.199406 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.199086    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-main-tls\") pod \"987de4a4-9a5a-4a49-841c-94efbff7380a\" (UID: \"987de4a4-9a5a-4a49-841c-94efbff7380a\") "
Sep 16 18:49:03.199406 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.199154    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:03.199406 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.199230    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:03.199839 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.199428    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-metrics-client-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.199839 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.199459    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.201363 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.201331    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:03.201989 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.201964    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:03.201989 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.201964    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-config-out" (OuterVolumeSpecName: "config-out") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:03.202246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.202025    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-config-volume" (OuterVolumeSpecName: "config-volume") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:03.202313 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.202255    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-kube-api-access-cfx2m" (OuterVolumeSpecName: "kube-api-access-cfx2m") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "kube-api-access-cfx2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:03.202313 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.202302    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:03.202495 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.202456    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:03.203740 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.203713    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:03.203888 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.203870    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:03.206274 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.206251    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:03.212061 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.212040    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-web-config" (OuterVolumeSpecName: "web-config") pod "987de4a4-9a5a-4a49-841c-94efbff7380a" (UID: "987de4a4-9a5a-4a49-841c-94efbff7380a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:03.300772 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300745    2567 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-cluster-tls-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.300772 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300768    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-config-out\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.300920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300778    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfx2m\" (UniqueName: \"kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-kube-api-access-cfx2m\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.300920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300787    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/987de4a4-9a5a-4a49-841c-94efbff7380a-tls-assets\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.300920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300795    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-web-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.300920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300804    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.300920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300812    2567 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.300920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300820    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/987de4a4-9a5a-4a49-841c-94efbff7380a-alertmanager-main-db\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.300920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300828    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.300920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300838    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-main-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.300920 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.300847    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/987de4a4-9a5a-4a49-841c-94efbff7380a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.312101 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.312073    2567 generic.go:356] "Generic (PLEG): container finished" podID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerID="eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e" exitCode=0
Sep 16 18:49:03.312101 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.312098    2567 generic.go:356] "Generic (PLEG): container finished" podID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerID="eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343" exitCode=0
Sep 16 18:49:03.312305 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.312139    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerDied","Data":"eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e"}
Sep 16 18:49:03.312305 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.312177    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.312305 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.312272    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerDied","Data":"eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343"}
Sep 16 18:49:03.312452 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.312321    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"987de4a4-9a5a-4a49-841c-94efbff7380a","Type":"ContainerDied","Data":"e1efefe03d58b6f0732964ed6770d4cfaee139fc3f485ba8bffca5d527841a7c"}
Sep 16 18:49:03.312452 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.312338    2567 scope.go:117] "RemoveContainer" containerID="15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486"
Sep 16 18:49:03.320709 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.320691    2567 scope.go:117] "RemoveContainer" containerID="eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e"
Sep 16 18:49:03.327175 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.327161    2567 scope.go:117] "RemoveContainer" containerID="be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b"
Sep 16 18:49:03.333736 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.333721    2567 scope.go:117] "RemoveContainer" containerID="eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343"
Sep 16 18:49:03.338209 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.338188    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:03.343247 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.343225    2567 scope.go:117] "RemoveContainer" containerID="c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2"
Sep 16 18:49:03.344936 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.344916    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:03.350382 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.350362    2567 scope.go:117] "RemoveContainer" containerID="ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de"
Sep 16 18:49:03.356942 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.356924    2567 scope.go:117] "RemoveContainer" containerID="508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522"
Sep 16 18:49:03.363613 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.363597    2567 scope.go:117] "RemoveContainer" containerID="15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486"
Sep 16 18:49:03.363899 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:49:03.363880    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486\": container with ID starting with 15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486 not found: ID does not exist" containerID="15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486"
Sep 16 18:49:03.363944 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.363909    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486"} err="failed to get container status \"15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486\": rpc error: code = NotFound desc = could not find container \"15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486\": container with ID starting with 15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486 not found: ID does not exist"
Sep 16 18:49:03.363944 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.363928    2567 scope.go:117] "RemoveContainer" containerID="eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e"
Sep 16 18:49:03.364214 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:49:03.364197    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e\": container with ID starting with eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e not found: ID does not exist" containerID="eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e"
Sep 16 18:49:03.364252 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.364220    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e"} err="failed to get container status \"eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e\": rpc error: code = NotFound desc = could not find container \"eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e\": container with ID starting with eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e not found: ID does not exist"
Sep 16 18:49:03.364252 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.364236    2567 scope.go:117] "RemoveContainer" containerID="be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b"
Sep 16 18:49:03.364464 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:49:03.364449    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b\": container with ID starting with be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b not found: ID does not exist" containerID="be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b"
Sep 16 18:49:03.364505 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.364468    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b"} err="failed to get container status \"be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b\": rpc error: code = NotFound desc = could not find container \"be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b\": container with ID starting with be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b not found: ID does not exist"
Sep 16 18:49:03.364505 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.364481    2567 scope.go:117] "RemoveContainer" containerID="eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343"
Sep 16 18:49:03.364718 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:49:03.364696    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343\": container with ID starting with eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343 not found: ID does not exist" containerID="eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343"
Sep 16 18:49:03.364761 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.364725    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343"} err="failed to get container status \"eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343\": rpc error: code = NotFound desc = could not find container \"eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343\": container with ID starting with eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343 not found: ID does not exist"
Sep 16 18:49:03.364761 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.364740    2567 scope.go:117] "RemoveContainer" containerID="c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2"
Sep 16 18:49:03.364965 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:49:03.364948    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2\": container with ID starting with c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2 not found: ID does not exist" containerID="c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2"
Sep 16 18:49:03.365008 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.364970    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2"} err="failed to get container status \"c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2\": rpc error: code = NotFound desc = could not find container \"c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2\": container with ID starting with c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2 not found: ID does not exist"
Sep 16 18:49:03.365008 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.364985    2567 scope.go:117] "RemoveContainer" containerID="ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de"
Sep 16 18:49:03.365234 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:49:03.365219    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de\": container with ID starting with ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de not found: ID does not exist" containerID="ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de"
Sep 16 18:49:03.365277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.365238    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de"} err="failed to get container status \"ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de\": rpc error: code = NotFound desc = could not find container \"ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de\": container with ID starting with ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de not found: ID does not exist"
Sep 16 18:49:03.365277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.365250    2567 scope.go:117] "RemoveContainer" containerID="508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522"
Sep 16 18:49:03.365481 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:49:03.365466    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522\": container with ID starting with 508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522 not found: ID does not exist" containerID="508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522"
Sep 16 18:49:03.365524 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.365485    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522"} err="failed to get container status \"508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522\": rpc error: code = NotFound desc = could not find container \"508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522\": container with ID starting with 508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522 not found: ID does not exist"
Sep 16 18:49:03.365524 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.365497    2567 scope.go:117] "RemoveContainer" containerID="15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486"
Sep 16 18:49:03.365717 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.365701    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486"} err="failed to get container status \"15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486\": rpc error: code = NotFound desc = could not find container \"15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486\": container with ID starting with 15d4eb987debcb405a9682a5510ee120a28a32fd0e9d40b53b7bd898cb594486 not found: ID does not exist"
Sep 16 18:49:03.365764 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.365718    2567 scope.go:117] "RemoveContainer" containerID="eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e"
Sep 16 18:49:03.365932 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.365917    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e"} err="failed to get container status \"eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e\": rpc error: code = NotFound desc = could not find container \"eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e\": container with ID starting with eb7cae4a4405f5fabaabc6febf6b30c3950ed38f1163fcc5a98170daee5f524e not found: ID does not exist"
Sep 16 18:49:03.365979 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.365932    2567 scope.go:117] "RemoveContainer" containerID="be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b"
Sep 16 18:49:03.366166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.366147    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b"} err="failed to get container status \"be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b\": rpc error: code = NotFound desc = could not find container \"be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b\": container with ID starting with be32e47dc0d595a6b550c72b292962d732e110ec2c3d96d269fe6e9217215d3b not found: ID does not exist"
Sep 16 18:49:03.366222 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.366167    2567 scope.go:117] "RemoveContainer" containerID="eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343"
Sep 16 18:49:03.366374 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.366359    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343"} err="failed to get container status \"eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343\": rpc error: code = NotFound desc = could not find container \"eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343\": container with ID starting with eaa061005967fbd7ebc07a55b968ca08ded42de5a854de0d72acacb216886343 not found: ID does not exist"
Sep 16 18:49:03.366413 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.366374    2567 scope.go:117] "RemoveContainer" containerID="c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2"
Sep 16 18:49:03.366583 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.366569    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2"} err="failed to get container status \"c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2\": rpc error: code = NotFound desc = could not find container \"c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2\": container with ID starting with c7701dac3ebdd6c96a42f06bbd949f2cdb0f26d4fc0cafccbf7075f75a2287d2 not found: ID does not exist"
Sep 16 18:49:03.366629 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.366583    2567 scope.go:117] "RemoveContainer" containerID="ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de"
Sep 16 18:49:03.366820 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.366799    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de"} err="failed to get container status \"ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de\": rpc error: code = NotFound desc = could not find container \"ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de\": container with ID starting with ac8ec80d396626cdac7308227b63a4e6f5b54b6fe2e885167811e50c80d414de not found: ID does not exist"
Sep 16 18:49:03.366820 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.366819    2567 scope.go:117] "RemoveContainer" containerID="508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522"
Sep 16 18:49:03.367120 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.367081    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522"} err="failed to get container status \"508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522\": rpc error: code = NotFound desc = could not find container \"508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522\": container with ID starting with 508246a2209f9f75ecb00a0ec19264c98fa34196bb46f4b38899834f71ac8522 not found: ID does not exist"
Sep 16 18:49:03.381626 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381592    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:03.381928 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381915    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="config-reloader"
Sep 16 18:49:03.381970 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381931    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="config-reloader"
Sep 16 18:49:03.381970 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381940    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" containerName="console"
Sep 16 18:49:03.381970 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381945    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" containerName="console"
Sep 16 18:49:03.381970 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381957    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy-metric"
Sep 16 18:49:03.381970 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381962    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy-metric"
Sep 16 18:49:03.381970 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381970    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a8d8757-4193-4971-a87a-2dc7b8ba30fe" containerName="console"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381976    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a8d8757-4193-4971-a87a-2dc7b8ba30fe" containerName="console"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381982    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy-web"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.381989    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy-web"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382000    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="alertmanager"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382006    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="alertmanager"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382012    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d974f52-86bb-420f-a91d-39f2dfa2c62b" containerName="registry"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382018    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d974f52-86bb-420f-a91d-39f2dfa2c62b" containerName="registry"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382024    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="init-config-reloader"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382028    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="init-config-reloader"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382035    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="prom-label-proxy"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382040    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="prom-label-proxy"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382050    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382054    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382128    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a8d8757-4193-4971-a87a-2dc7b8ba30fe" containerName="console"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382136    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d974f52-86bb-420f-a91d-39f2dfa2c62b" containerName="registry"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382144    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="alertmanager"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382156    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382164    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49eb2ae-8b1e-4fdd-b15a-f783cc63c3f8" containerName="console"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382169    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy-metric"
Sep 16 18:49:03.382166 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382174    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="config-reloader"
Sep 16 18:49:03.382711 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382179    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="kube-rbac-proxy-web"
Sep 16 18:49:03.382711 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.382186    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" containerName="prom-label-proxy"
Sep 16 18:49:03.387419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.387402    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.393481 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.393460    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:49:03.393481 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.393460    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:49:03.395048 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.395031    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:49:03.395048 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.395040    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:49:03.395234 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.395221    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:49:03.395335 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.395317    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:49:03.395428 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.395411    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lzdgq\""
Sep 16 18:49:03.396285 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.396270    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:49:03.396558 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.396546    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:49:03.411935 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.411912    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:03.413674 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.413652    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:49:03.503461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503379    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503435    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503461 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503462    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-web-config\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503681 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503681 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503540    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503681 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503561    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503681 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503584    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-config-out\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503806 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503673    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503806 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lq9n\" (UniqueName: \"kubernetes.io/projected/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-kube-api-access-8lq9n\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503806 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503760    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503806 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503775    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503806 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503795    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.503944 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.503829    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604647 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604606    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604647 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604649    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-web-config\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604681    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604793    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604822    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604995 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604848    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-config-out\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604995 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604874    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604995 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604903    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lq9n\" (UniqueName: \"kubernetes.io/projected/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-kube-api-access-8lq9n\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604995 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604940    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604995 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604963    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.604995 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.604989    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.605381 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.605039    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.605381 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.605082    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.605873 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.605607    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.606001 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.605905    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.606514 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.606483    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.607847 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.607814    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.608035 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.608009    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.608201 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.608177    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-web-config\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.608287 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.608273    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.608418 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.608386    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-config-out\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.608538 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.608520    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.608605 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.608590    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.608694 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.608678    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.609284 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.609266    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.614058 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.614042    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lq9n\" (UniqueName: \"kubernetes.io/projected/e96dfc47-4e0f-4a77-8e65-1282b08a22ee-kube-api-access-8lq9n\") pod \"alertmanager-main-0\" (UID: \"e96dfc47-4e0f-4a77-8e65-1282b08a22ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.696129 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.696089    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:03.978507 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:03.978332    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:03.980926 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:49:03.980892    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96dfc47_4e0f_4a77_8e65_1282b08a22ee.slice/crio-d187c960f2c4fcc8d4d0e2cc65c54b309565dee02c545b5ab4e6c9ca26f09e6d WatchSource:0}: Error finding container d187c960f2c4fcc8d4d0e2cc65c54b309565dee02c545b5ab4e6c9ca26f09e6d: Status 404 returned error can't find the container with id d187c960f2c4fcc8d4d0e2cc65c54b309565dee02c545b5ab4e6c9ca26f09e6d
Sep 16 18:49:04.319149 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:04.319054    2567 generic.go:356] "Generic (PLEG): container finished" podID="e96dfc47-4e0f-4a77-8e65-1282b08a22ee" containerID="df32d2b871a8ca3746a191faf62afe970d3645c46c722c0cf198cd9fd416ce49" exitCode=0
Sep 16 18:49:04.319459 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:04.319147    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e96dfc47-4e0f-4a77-8e65-1282b08a22ee","Type":"ContainerDied","Data":"df32d2b871a8ca3746a191faf62afe970d3645c46c722c0cf198cd9fd416ce49"}
Sep 16 18:49:04.319459 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:04.319228    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e96dfc47-4e0f-4a77-8e65-1282b08a22ee","Type":"ContainerStarted","Data":"d187c960f2c4fcc8d4d0e2cc65c54b309565dee02c545b5ab4e6c9ca26f09e6d"}
Sep 16 18:49:05.326258 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.326172    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e96dfc47-4e0f-4a77-8e65-1282b08a22ee","Type":"ContainerStarted","Data":"7bd526863445c1b5a52fdc5e3d96934ddac5b42def7a3e3f5668f9c2d371f19a"}
Sep 16 18:49:05.326585 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.326259    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e96dfc47-4e0f-4a77-8e65-1282b08a22ee","Type":"ContainerStarted","Data":"c6b74c2ec4991fa3acb99bfdfb327c6cc0d91d23fb3052bdb808f3f776f4ab23"}
Sep 16 18:49:05.326585 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.326286    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e96dfc47-4e0f-4a77-8e65-1282b08a22ee","Type":"ContainerStarted","Data":"f3342ed6040448491409b70826afd8d75c079704c093a7b9f67dcf98149fe7b0"}
Sep 16 18:49:05.326585 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.326310    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e96dfc47-4e0f-4a77-8e65-1282b08a22ee","Type":"ContainerStarted","Data":"9bb57e23bda66a3d04524e3ecf98fe678a112b28785c0e1a58ee21984a5a192c"}
Sep 16 18:49:05.326585 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.326334    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e96dfc47-4e0f-4a77-8e65-1282b08a22ee","Type":"ContainerStarted","Data":"b0b2b592526baa87674dcb4bb51fc8a57195069afc1af5bd00c38b3d23a8182e"}
Sep 16 18:49:05.326585 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.326355    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e96dfc47-4e0f-4a77-8e65-1282b08a22ee","Type":"ContainerStarted","Data":"77d4826e742ddccda445b7183e3646e835b520e8b76e14a6627812cb12c05d9a"}
Sep 16 18:49:05.333412 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.333387    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987de4a4-9a5a-4a49-841c-94efbff7380a" path="/var/lib/kubelet/pods/987de4a4-9a5a-4a49-841c-94efbff7380a/volumes"
Sep 16 18:49:05.359021 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.358969    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.344847883 podStartE2EDuration="2.358949363s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.10148ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:05.357800732 +0000 UTC m=+212.546247415" watchObservedRunningTime="2026-09-16 18:49:05.358949363 +0000 UTC m=+212.547396036"
Sep 16 18:49:05.846092 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.846056    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"]
Sep 16 18:49:05.850475 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.850449    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:05.853323 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.853295    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:49:05.853444 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.853334    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:49:05.853842 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.853823    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:49:05.854031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.854007    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jgndz\""
Sep 16 18:49:05.854228 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.854190    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:49:05.854310 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.854194    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:49:05.859635 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.859616    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:49:05.872031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:05.872002    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"]
Sep 16 18:49:06.026767 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.026730    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ea8c55-34c4-4860-b8a3-c601090ef138-serving-certs-ca-bundle\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.026767 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.026766    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-secret-telemeter-client\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.026965 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.026785    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-federate-client-tls\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.026965 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.026826    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ea8c55-34c4-4860-b8a3-c601090ef138-metrics-client-ca\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.026965 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.026884    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-telemeter-client-tls\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.026965 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.026921    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.027086 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.027001    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ea8c55-34c4-4860-b8a3-c601090ef138-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.027086 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.027046    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mdbk\" (UniqueName: \"kubernetes.io/projected/a1ea8c55-34c4-4860-b8a3-c601090ef138-kube-api-access-6mdbk\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.127614 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.127587    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.127741 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.127633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ea8c55-34c4-4860-b8a3-c601090ef138-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.127741 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.127658    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mdbk\" (UniqueName: \"kubernetes.io/projected/a1ea8c55-34c4-4860-b8a3-c601090ef138-kube-api-access-6mdbk\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.127741 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.127682    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ea8c55-34c4-4860-b8a3-c601090ef138-serving-certs-ca-bundle\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.127741 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.127700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-secret-telemeter-client\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.127741 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.127716    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-federate-client-tls\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.127741 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.127733    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ea8c55-34c4-4860-b8a3-c601090ef138-metrics-client-ca\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.127993 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.127874    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-telemeter-client-tls\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.128537 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.128510    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ea8c55-34c4-4860-b8a3-c601090ef138-metrics-client-ca\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.128670 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.128566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ea8c55-34c4-4860-b8a3-c601090ef138-serving-certs-ca-bundle\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.128732 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.128691    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ea8c55-34c4-4860-b8a3-c601090ef138-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.130362 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.130337    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-telemeter-client-tls\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.130474 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.130432    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.130526 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.130496    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-federate-client-tls\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.130570 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.130553    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1ea8c55-34c4-4860-b8a3-c601090ef138-secret-telemeter-client\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.137170 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.137091    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mdbk\" (UniqueName: \"kubernetes.io/projected/a1ea8c55-34c4-4860-b8a3-c601090ef138-kube-api-access-6mdbk\") pod \"telemeter-client-7bddd84c9-gvjhh\" (UID: \"a1ea8c55-34c4-4860-b8a3-c601090ef138\") " pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.160711 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.160681    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:06.160811 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.160795    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"
Sep 16 18:49:06.161049 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.161013    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="prometheus" containerID="cri-o://70dbd150ed3460c4b22fb06d4549adb4a0e82a3c74928efe8c6743a8b086e57d" gracePeriod=600
Sep 16 18:49:06.161103 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.161061    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bceee3375960bc42b65525334f59c846bdf5f106946e70a1fb5512323b6fe71e" gracePeriod=600
Sep 16 18:49:06.161176 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.161095    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="thanos-sidecar" containerID="cri-o://cef259f8f623d3166d377d23dcef060ee65ba2aaed7cc57c8dfdeeb19593c351" gracePeriod=600
Sep 16 18:49:06.161176 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.161127    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy" containerID="cri-o://71edf8732ceccf35227a0bc5729fb7792eab04c64fafdd4dc96fb4b0701f61b0" gracePeriod=600
Sep 16 18:49:06.161176 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.161099    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy-web" containerID="cri-o://276f062aff5f5007c31890c992af0023e10b5235da1d0ae6ad636dba29b290e0" gracePeriod=600
Sep 16 18:49:06.161176 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.161140    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="config-reloader" containerID="cri-o://355812060b7b89f38af2a71b0e1041bb795e9794a97e996ecac87cd62a61070d" gracePeriod=600
Sep 16 18:49:06.334314 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.334286    2567 generic.go:356] "Generic (PLEG): container finished" podID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerID="bceee3375960bc42b65525334f59c846bdf5f106946e70a1fb5512323b6fe71e" exitCode=0
Sep 16 18:49:06.334314 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.334310    2567 generic.go:356] "Generic (PLEG): container finished" podID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerID="71edf8732ceccf35227a0bc5729fb7792eab04c64fafdd4dc96fb4b0701f61b0" exitCode=0
Sep 16 18:49:06.334314 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.334316    2567 generic.go:356] "Generic (PLEG): container finished" podID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerID="cef259f8f623d3166d377d23dcef060ee65ba2aaed7cc57c8dfdeeb19593c351" exitCode=0
Sep 16 18:49:06.334740 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.334323    2567 generic.go:356] "Generic (PLEG): container finished" podID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerID="355812060b7b89f38af2a71b0e1041bb795e9794a97e996ecac87cd62a61070d" exitCode=0
Sep 16 18:49:06.334740 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.334329    2567 generic.go:356] "Generic (PLEG): container finished" podID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerID="70dbd150ed3460c4b22fb06d4549adb4a0e82a3c74928efe8c6743a8b086e57d" exitCode=0
Sep 16 18:49:06.334740 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.334354    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerDied","Data":"bceee3375960bc42b65525334f59c846bdf5f106946e70a1fb5512323b6fe71e"}
Sep 16 18:49:06.334740 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.334421    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerDied","Data":"71edf8732ceccf35227a0bc5729fb7792eab04c64fafdd4dc96fb4b0701f61b0"}
Sep 16 18:49:06.334740 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.334449    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerDied","Data":"cef259f8f623d3166d377d23dcef060ee65ba2aaed7cc57c8dfdeeb19593c351"}
Sep 16 18:49:06.334740 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.334476    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerDied","Data":"355812060b7b89f38af2a71b0e1041bb795e9794a97e996ecac87cd62a61070d"}
Sep 16 18:49:06.334740 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.334504    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerDied","Data":"70dbd150ed3460c4b22fb06d4549adb4a0e82a3c74928efe8c6743a8b086e57d"}
Sep 16 18:49:06.448978 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:06.448955    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bddd84c9-gvjhh"]
Sep 16 18:49:06.451399 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:49:06.451371    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ea8c55_34c4_4860_b8a3_c601090ef138.slice/crio-99526eeabe7a4d684ce297a484ddde1230d1268871917af54b9d36222310c593 WatchSource:0}: Error finding container 99526eeabe7a4d684ce297a484ddde1230d1268871917af54b9d36222310c593: Status 404 returned error can't find the container with id 99526eeabe7a4d684ce297a484ddde1230d1268871917af54b9d36222310c593
Sep 16 18:49:07.351470 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.351420    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh" event={"ID":"a1ea8c55-34c4-4860-b8a3-c601090ef138","Type":"ContainerStarted","Data":"99526eeabe7a4d684ce297a484ddde1230d1268871917af54b9d36222310c593"}
Sep 16 18:49:07.356299 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.356129    2567 generic.go:356] "Generic (PLEG): container finished" podID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerID="276f062aff5f5007c31890c992af0023e10b5235da1d0ae6ad636dba29b290e0" exitCode=0
Sep 16 18:49:07.356299 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.356146    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerDied","Data":"276f062aff5f5007c31890c992af0023e10b5235da1d0ae6ad636dba29b290e0"}
Sep 16 18:49:07.431038 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.431011    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:07.543601 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543573    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-metrics-client-certs\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.543765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543631    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-trusted-ca-bundle\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.543765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543651    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-serving-certs-ca-bundle\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.543765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543671    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-2\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.543765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543687    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.543765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543703    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-db\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.543765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543733    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-grpc-tls\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.543765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543763    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-1\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544152 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543787    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-metrics-client-ca\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544152 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543818    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-kubelet-serving-ca-bundle\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544152 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543886    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-0\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544152 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.543912    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-thanos-prometheus-http-client-file\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544152 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544130    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-web-config\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544166    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:07.544398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544195    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544229    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config-out\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544259    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-tls-assets\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544284    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2pff\" (UniqueName: \"kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-kube-api-access-w2pff\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544315    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-kube-rbac-proxy\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544361    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-tls\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544394    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\" (UID: \"5ff00e73-8b62-4397-bb86-4f6fba1a9842\") "
Sep 16 18:49:07.544796 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544697    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.545197 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544191    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:07.545197 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544563    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:07.545367 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.544931    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:07.545367 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.545033    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:07.545367 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.545160    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:07.545840 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.545618    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:07.546767 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.546731    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:07.547264 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.547234    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:07.547768 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.547738    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:07.548032 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.547991    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-kube-api-access-w2pff" (OuterVolumeSpecName: "kube-api-access-w2pff") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "kube-api-access-w2pff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:07.548490 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.548448    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config" (OuterVolumeSpecName: "config") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:07.549082 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.549038    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:07.549082 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.549045    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:07.549231 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.549078    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:07.549300 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.549283    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:07.549639 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.549619    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:07.549755 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.549737    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config-out" (OuterVolumeSpecName: "config-out") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:07.550817 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.550787    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:07.558359 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.558339    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-web-config" (OuterVolumeSpecName: "web-config") pod "5ff00e73-8b62-4397-bb86-4f6fba1a9842" (UID: "5ff00e73-8b62-4397-bb86-4f6fba1a9842"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:07.646185 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646145    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646185 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646188    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646204    2567 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-metrics-client-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646219    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646236    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646250    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646265    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-db\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646279    2567 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-grpc-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646292    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646305    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-metrics-client-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646320    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646334    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ff00e73-8b62-4397-bb86-4f6fba1a9842-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646348    2567 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646363    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-web-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646376    2567 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646388    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ff00e73-8b62-4397-bb86-4f6fba1a9842-config-out\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646400    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-tls-assets\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646412    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2pff\" (UniqueName: \"kubernetes.io/projected/5ff00e73-8b62-4397-bb86-4f6fba1a9842-kube-api-access-w2pff\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:07.646419 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:07.646426    2567 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ff00e73-8b62-4397-bb86-4f6fba1a9842-secret-kube-rbac-proxy\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:08.361865 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.361837    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh" event={"ID":"a1ea8c55-34c4-4860-b8a3-c601090ef138","Type":"ContainerStarted","Data":"aa38118a5163ae5bfc557e9d778e5065dee47f1465e1ff98b472001aba0a8a13"}
Sep 16 18:49:08.362224 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.361908    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh" event={"ID":"a1ea8c55-34c4-4860-b8a3-c601090ef138","Type":"ContainerStarted","Data":"dc6644a2ba454124e51b70bf2419f9449bb55d451c9bf9873985e082aced2abb"}
Sep 16 18:49:08.365743 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.365719    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ff00e73-8b62-4397-bb86-4f6fba1a9842","Type":"ContainerDied","Data":"6b2f31008da936448755a1987ecabbc9bd3de4974dec6f207cae76d972846ba9"}
Sep 16 18:49:08.365825 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.365808    2567 scope.go:117] "RemoveContainer" containerID="bceee3375960bc42b65525334f59c846bdf5f106946e70a1fb5512323b6fe71e"
Sep 16 18:49:08.365896 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.365728    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.376347 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.376323    2567 scope.go:117] "RemoveContainer" containerID="71edf8732ceccf35227a0bc5729fb7792eab04c64fafdd4dc96fb4b0701f61b0"
Sep 16 18:49:08.384227 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.384206    2567 scope.go:117] "RemoveContainer" containerID="276f062aff5f5007c31890c992af0023e10b5235da1d0ae6ad636dba29b290e0"
Sep 16 18:49:08.391029 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.391013    2567 scope.go:117] "RemoveContainer" containerID="cef259f8f623d3166d377d23dcef060ee65ba2aaed7cc57c8dfdeeb19593c351"
Sep 16 18:49:08.396742 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.396717    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:08.400533 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.400513    2567 scope.go:117] "RemoveContainer" containerID="355812060b7b89f38af2a71b0e1041bb795e9794a97e996ecac87cd62a61070d"
Sep 16 18:49:08.400691 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.400666    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:08.408063 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.407958    2567 scope.go:117] "RemoveContainer" containerID="70dbd150ed3460c4b22fb06d4549adb4a0e82a3c74928efe8c6743a8b086e57d"
Sep 16 18:49:08.416909 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.416888    2567 scope.go:117] "RemoveContainer" containerID="bdc92997f2c8980c94eb45352781347e31c82e609e2e20e6512441b8cc4e6a1b"
Sep 16 18:49:08.435548 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435523    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:08.435832 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435818    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy"
Sep 16 18:49:08.435871 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435837    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy"
Sep 16 18:49:08.435871 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435852    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="init-config-reloader"
Sep 16 18:49:08.435871 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435858    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="init-config-reloader"
Sep 16 18:49:08.435871 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435864    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="thanos-sidecar"
Sep 16 18:49:08.435871 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435869    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="thanos-sidecar"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435878    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="config-reloader"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435883    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="config-reloader"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435893    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy-thanos"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435898    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy-thanos"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435907    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="prometheus"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435912    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="prometheus"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435919    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy-web"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435924    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy-web"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.435996    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy-web"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.436005    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="prometheus"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.436013    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="thanos-sidecar"
Sep 16 18:49:08.436018 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.436019    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy-thanos"
Sep 16 18:49:08.436361 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.436024    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="config-reloader"
Sep 16 18:49:08.436361 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.436028    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" containerName="kube-rbac-proxy"
Sep 16 18:49:08.441262 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.441238    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.444855 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.444837    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:49:08.444977 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.444962    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:49:08.445808 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.445793    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:49:08.445941 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.445925    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:49:08.445987 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.445957    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n5hnw\""
Sep 16 18:49:08.449675 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.449652    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:49:08.449859 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.449836    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:49:08.449946 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.449839    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:49:08.449946 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.449842    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:49:08.450131 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.449890    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:49:08.450131 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.449891    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:49:08.450277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.450262    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8bj3qtqeotiuf\""
Sep 16 18:49:08.451186 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.451170    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:49:08.452001 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.451980    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:49:08.452839 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.452823    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:49:08.480634 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.480607    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:49:08.480767 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.480714    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:08.554695 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.554661    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-config\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.554695 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.554708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83be31a6-f749-46c3-99d5-97c37a045d77-config-out\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.554933 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.554732    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.554933 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.554759    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-web-config\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.554933 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.554782    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.554933 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.554812    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.554933 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.554835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.554933 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.554875    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.554933 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.554901    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83be31a6-f749-46c3-99d5-97c37a045d77-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.554967    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.555027    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.555061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.555081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.555152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.555181    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.555203    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.555219    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555480 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.555263    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555480 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.555282    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6vx\" (UniqueName: \"kubernetes.io/projected/83be31a6-f749-46c3-99d5-97c37a045d77-kube-api-access-7g6vx\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.555480 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.555319    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.656384 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656315    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.656384 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.656384 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656377    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.656655 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656396    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.656655 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656523    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.656655 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.656655 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656583    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.656859 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656763    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.656859 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656813    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.656859 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656843    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6vx\" (UniqueName: \"kubernetes.io/projected/83be31a6-f749-46c3-99d5-97c37a045d77-kube-api-access-7g6vx\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657001 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657001 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656910    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-config\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657001 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656940    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83be31a6-f749-46c3-99d5-97c37a045d77-config-out\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657001 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657001 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.656986    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-web-config\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657271 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.657009    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657271 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.657043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657271 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.657068    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657271 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.657100    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657271 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.657151    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83be31a6-f749-46c3-99d5-97c37a045d77-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.657531 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.657295    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.659254 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.659227    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.659520 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.659495    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.659800 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.659773    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83be31a6-f749-46c3-99d5-97c37a045d77-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.659884 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.659800    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.660292 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.660270    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83be31a6-f749-46c3-99d5-97c37a045d77-config-out\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.660292 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.660285    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.660543 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.660523    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.660605 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.660546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.660605 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.660563    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.660605 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.660524    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.661012 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.660804    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.661207 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.661180    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be31a6-f749-46c3-99d5-97c37a045d77-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.661326 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.661305    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.661708 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.661689    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.662321 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.662297    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.662943 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.662918    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-web-config\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.663159 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.663140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.663210 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.663173    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/83be31a6-f749-46c3-99d5-97c37a045d77-config\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.670582 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.670563    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6vx\" (UniqueName: \"kubernetes.io/projected/83be31a6-f749-46c3-99d5-97c37a045d77-kube-api-access-7g6vx\") pod \"prometheus-k8s-0\" (UID: \"83be31a6-f749-46c3-99d5-97c37a045d77\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:08.751552 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:08.751525    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:09.040981 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.040955    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:09.042940 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:49:09.042915    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83be31a6_f749_46c3_99d5_97c37a045d77.slice/crio-4dace81b57ae3ede77353d3a0f9a0816275f5a2ecaaae64bf8d059c0663e4035 WatchSource:0}: Error finding container 4dace81b57ae3ede77353d3a0f9a0816275f5a2ecaaae64bf8d059c0663e4035: Status 404 returned error can't find the container with id 4dace81b57ae3ede77353d3a0f9a0816275f5a2ecaaae64bf8d059c0663e4035
Sep 16 18:49:09.333151 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.333053    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff00e73-8b62-4397-bb86-4f6fba1a9842" path="/var/lib/kubelet/pods/5ff00e73-8b62-4397-bb86-4f6fba1a9842/volumes"
Sep 16 18:49:09.372462 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.372425    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh" event={"ID":"a1ea8c55-34c4-4860-b8a3-c601090ef138","Type":"ContainerStarted","Data":"04f1eb4b6384d2ff5780adb4f832e13a3b0f0f8d995147511d23dc90dfd7ad12"}
Sep 16 18:49:09.373828 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.373802    2567 generic.go:356] "Generic (PLEG): container finished" podID="83be31a6-f749-46c3-99d5-97c37a045d77" containerID="d38ae792dd3d43bd0c560715b93ca33601a1d3f581ed65d84f8ac63b1f97a5be" exitCode=0
Sep 16 18:49:09.373944 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.373898    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be31a6-f749-46c3-99d5-97c37a045d77","Type":"ContainerDied","Data":"d38ae792dd3d43bd0c560715b93ca33601a1d3f581ed65d84f8ac63b1f97a5be"}
Sep 16 18:49:09.374003 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.373955    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be31a6-f749-46c3-99d5-97c37a045d77","Type":"ContainerStarted","Data":"4dace81b57ae3ede77353d3a0f9a0816275f5a2ecaaae64bf8d059c0663e4035"}
Sep 16 18:49:09.408397 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.408359    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7bddd84c9-gvjhh" podStartSLOduration=2.651176255 podStartE2EDuration="4.40834558s" totalImagesPullingTime="1.757169325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:09.406875386 +0000 UTC m=+216.595322081" watchObservedRunningTime="2026-09-16 18:49:09.40834558 +0000 UTC m=+216.596792261"
Sep 16 18:49:09.507953 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.507926    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f46cb85bc-gj9bv"]
Sep 16 18:49:09.512861 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.512702    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.534603 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.534007    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f46cb85bc-gj9bv"]
Sep 16 18:49:09.665684 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.665647    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-trusted-ca-bundle\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.665684 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.665681    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-oauth-serving-cert\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.665845 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.665717    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-config\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.665845 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.665793    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-service-ca\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.665845 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.665830    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knnwc\" (UniqueName: \"kubernetes.io/projected/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-kube-api-access-knnwc\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.665993 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.665861    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-oauth-config\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.665993 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.665960    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-serving-cert\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.767138 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.767095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-service-ca\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.767261 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.767146    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knnwc\" (UniqueName: \"kubernetes.io/projected/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-kube-api-access-knnwc\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.767261 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.767171    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-oauth-config\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.767261 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.767194    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-serving-cert\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.767261 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.767225    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-trusted-ca-bundle\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.767261 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.767249    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-oauth-serving-cert\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.767589 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.767308    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-config\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.768157 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.767835    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-service-ca\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.768157 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.767963    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-oauth-serving-cert\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.768157 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.768077    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-trusted-ca-bundle\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.768348 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.768175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-config\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.769986 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.769966    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-serving-cert\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.770085 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.770069    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-oauth-config\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.776550 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.776533    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knnwc\" (UniqueName: \"kubernetes.io/projected/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-kube-api-access-knnwc\") pod \"console-f46cb85bc-gj9bv\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") " pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:09.828268 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:09.828241    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:10.100772 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.100753    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f46cb85bc-gj9bv"]
Sep 16 18:49:10.102909 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:49:10.102884    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb016dc4_87d2_4ddf_86a9_0b1a90667e50.slice/crio-5a1e75b45a828a2788a66a11fc14bd8ed27462dea1057266edc492f2d9a0373d WatchSource:0}: Error finding container 5a1e75b45a828a2788a66a11fc14bd8ed27462dea1057266edc492f2d9a0373d: Status 404 returned error can't find the container with id 5a1e75b45a828a2788a66a11fc14bd8ed27462dea1057266edc492f2d9a0373d
Sep 16 18:49:10.380670 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.380638    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f46cb85bc-gj9bv" event={"ID":"eb016dc4-87d2-4ddf-86a9-0b1a90667e50","Type":"ContainerStarted","Data":"ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1"}
Sep 16 18:49:10.381076 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.380695    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f46cb85bc-gj9bv" event={"ID":"eb016dc4-87d2-4ddf-86a9-0b1a90667e50","Type":"ContainerStarted","Data":"5a1e75b45a828a2788a66a11fc14bd8ed27462dea1057266edc492f2d9a0373d"}
Sep 16 18:49:10.384189 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.384160    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be31a6-f749-46c3-99d5-97c37a045d77","Type":"ContainerStarted","Data":"cf865e6e44cff600b3d94572d82afcd3237c81d65bc1157c3d81bcb3270688f7"}
Sep 16 18:49:10.384316 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.384230    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be31a6-f749-46c3-99d5-97c37a045d77","Type":"ContainerStarted","Data":"0a36f2b993ed9bc99c293b17fa712599ff767ce278e4fbf194bde421c4061926"}
Sep 16 18:49:10.384316 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.384274    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be31a6-f749-46c3-99d5-97c37a045d77","Type":"ContainerStarted","Data":"bd909e2a71b0af4d2e2d1e33a9c356c6d187944943dca1f9fed9be0c363919a3"}
Sep 16 18:49:10.384444 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.384316    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be31a6-f749-46c3-99d5-97c37a045d77","Type":"ContainerStarted","Data":"80aafb746a6eb99478efa76d233b54a6bf1a845d66e6c48084af8bd7ba1979bc"}
Sep 16 18:49:10.384444 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.384353    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be31a6-f749-46c3-99d5-97c37a045d77","Type":"ContainerStarted","Data":"36924ac1240484cd0fd1536f189299adbe8e4e4e1ae2df91abebae880bf1638a"}
Sep 16 18:49:10.384444 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.384396    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be31a6-f749-46c3-99d5-97c37a045d77","Type":"ContainerStarted","Data":"1aa54ed8f1775e71254944a77564653b99f719aa37cb210960618de95a7f4ad4"}
Sep 16 18:49:10.404274 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.404229    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f46cb85bc-gj9bv" podStartSLOduration=1.404214497 podStartE2EDuration="1.404214497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:10.402493163 +0000 UTC m=+217.590939859" watchObservedRunningTime="2026-09-16 18:49:10.404214497 +0000 UTC m=+217.592661179"
Sep 16 18:49:10.442499 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:10.442458    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.426191369 podStartE2EDuration="2.442445106s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.253737ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:10.439636841 +0000 UTC m=+217.628083524" watchObservedRunningTime="2026-09-16 18:49:10.442445106 +0000 UTC m=+217.630891787"
Sep 16 18:49:13.751964 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:13.751932    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:19.829361 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:19.829333    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:19.829730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:19.829415    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:19.833958 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:19.833937    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:20.434778 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:20.434750    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:49:20.490445 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:20.490413    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f967777b9-5h7rt"]
Sep 16 18:49:45.517374 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.517316    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f967777b9-5h7rt" podUID="8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" containerName="console" containerID="cri-o://d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0" gracePeriod=15
Sep 16 18:49:45.751095 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.751072    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:49:45.885815 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.885787    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ttrq\" (UniqueName: \"kubernetes.io/projected/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-kube-api-access-2ttrq\") pod \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") "
Sep 16 18:49:45.885993 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.885829    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-config\") pod \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") "
Sep 16 18:49:45.885993 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.885862    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-oauth-config\") pod \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") "
Sep 16 18:49:45.885993 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.885938    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-trusted-ca-bundle\") pod \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") "
Sep 16 18:49:45.885993 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.885981    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-serving-cert\") pod \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") "
Sep 16 18:49:45.886233 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.886003    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-oauth-serving-cert\") pod \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") "
Sep 16 18:49:45.886233 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.886165    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-service-ca\") pod \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\" (UID: \"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91\") "
Sep 16 18:49:45.886342 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.886321    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-config" (OuterVolumeSpecName: "console-config") pod "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" (UID: "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:45.886446 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.886424    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" (UID: "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:45.886513 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.886420    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" (UID: "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:45.886598 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.886580    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-trusted-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:45.886650 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.886604    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-oauth-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:45.886650 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.886609    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-service-ca" (OuterVolumeSpecName: "service-ca") pod "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" (UID: "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:45.886650 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.886619    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:45.888139 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.888097    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" (UID: "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:45.888425 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.888405    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-kube-api-access-2ttrq" (OuterVolumeSpecName: "kube-api-access-2ttrq") pod "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" (UID: "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91"). InnerVolumeSpecName "kube-api-access-2ttrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:45.888425 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.888406    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" (UID: "8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:45.987629 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.987599    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-oauth-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:45.987629 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.987624    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-console-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:45.987629 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.987633    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:45.987799 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:45.987642    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ttrq\" (UniqueName: \"kubernetes.io/projected/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91-kube-api-access-2ttrq\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:49:46.542035 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:46.542005    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" containerID="d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0" exitCode=0
Sep 16 18:49:46.542474 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:46.542047    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f967777b9-5h7rt"
Sep 16 18:49:46.542474 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:46.542095    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f967777b9-5h7rt" event={"ID":"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91","Type":"ContainerDied","Data":"d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0"}
Sep 16 18:49:46.542474 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:46.542161    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f967777b9-5h7rt" event={"ID":"8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91","Type":"ContainerDied","Data":"a02e124e796afabd4267e83a96f26a1651d00bbb813be886538af9e93a736fa0"}
Sep 16 18:49:46.542474 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:46.542186    2567 scope.go:117] "RemoveContainer" containerID="d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0"
Sep 16 18:49:46.550681 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:46.550663    2567 scope.go:117] "RemoveContainer" containerID="d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0"
Sep 16 18:49:46.550957 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:49:46.550936    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0\": container with ID starting with d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0 not found: ID does not exist" containerID="d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0"
Sep 16 18:49:46.551024 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:46.550966    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0"} err="failed to get container status \"d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0\": rpc error: code = NotFound desc = could not find container \"d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0\": container with ID starting with d91d56943465b2bc4dd10c641459af374eb77f9fe4225168bae93f32462f6cf0 not found: ID does not exist"
Sep 16 18:49:46.564128 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:46.564092    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f967777b9-5h7rt"]
Sep 16 18:49:46.565495 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:46.565477    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f967777b9-5h7rt"]
Sep 16 18:49:47.333548 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:49:47.333515    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" path="/var/lib/kubelet/pods/8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91/volumes"
Sep 16 18:50:08.752457 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:08.752363    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:50:08.767946 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:08.767921    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:50:09.650814 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:09.650786    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:50:22.666216 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.666177    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b4b6db4d5-cvqm8"]
Sep 16 18:50:22.666798 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.666687    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" containerName="console"
Sep 16 18:50:22.666798 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.666707    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" containerName="console"
Sep 16 18:50:22.666798 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.666798    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a51d2e1-0496-4ea6-9f30-9c4a7ecf1d91" containerName="console"
Sep 16 18:50:22.669422 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.669401    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.685575 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.685552    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b4b6db4d5-cvqm8"]
Sep 16 18:50:22.807979 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.807940    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-serving-cert\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.807979 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.807973    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-service-ca\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.808242 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.807992    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-oauth-serving-cert\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.808242 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.808020    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-oauth-config\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.808242 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.808090    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xw62\" (UniqueName: \"kubernetes.io/projected/acf20a86-bdc2-45f3-9e01-86cfca2c5485-kube-api-access-7xw62\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.808242 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.808161    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-trusted-ca-bundle\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.808242 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.808225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-config\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.909473 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.909438    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-serving-cert\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.909473 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.909471    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-service-ca\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.909673 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.909493    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-oauth-serving-cert\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.909673 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.909516    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-oauth-config\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.909782 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.909659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xw62\" (UniqueName: \"kubernetes.io/projected/acf20a86-bdc2-45f3-9e01-86cfca2c5485-kube-api-access-7xw62\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.909782 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.909723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-trusted-ca-bundle\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.909885 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.909789    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-config\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.910241 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.910215    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-oauth-serving-cert\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.910354 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.910217    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-service-ca\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.910516 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.910493    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-config\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.910872 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.910854    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-trusted-ca-bundle\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.912559 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.912532    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-serving-cert\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.912642 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.912562    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-oauth-config\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.921368 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.921317    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xw62\" (UniqueName: \"kubernetes.io/projected/acf20a86-bdc2-45f3-9e01-86cfca2c5485-kube-api-access-7xw62\") pod \"console-5b4b6db4d5-cvqm8\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") " pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:22.978261 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:22.978235    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:23.259524 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:23.259501    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b4b6db4d5-cvqm8"]
Sep 16 18:50:23.261083 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:50:23.261059    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf20a86_bdc2_45f3_9e01_86cfca2c5485.slice/crio-74d814616d00414945f5f03a434b17687892c6265d900e0871123a4ac8e38a8a WatchSource:0}: Error finding container 74d814616d00414945f5f03a434b17687892c6265d900e0871123a4ac8e38a8a: Status 404 returned error can't find the container with id 74d814616d00414945f5f03a434b17687892c6265d900e0871123a4ac8e38a8a
Sep 16 18:50:23.699256 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:23.699221    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4b6db4d5-cvqm8" event={"ID":"acf20a86-bdc2-45f3-9e01-86cfca2c5485","Type":"ContainerStarted","Data":"629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11"}
Sep 16 18:50:23.699617 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:23.699274    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4b6db4d5-cvqm8" event={"ID":"acf20a86-bdc2-45f3-9e01-86cfca2c5485","Type":"ContainerStarted","Data":"74d814616d00414945f5f03a434b17687892c6265d900e0871123a4ac8e38a8a"}
Sep 16 18:50:23.727121 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:23.727067    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b4b6db4d5-cvqm8" podStartSLOduration=1.727052177 podStartE2EDuration="1.727052177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:23.725964611 +0000 UTC m=+290.914411293" watchObservedRunningTime="2026-09-16 18:50:23.727052177 +0000 UTC m=+290.915498930"
Sep 16 18:50:32.978679 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:32.978639    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:32.979087 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:32.978725    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:32.983264 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:32.983239    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:33.212894 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:33.212863    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 18:50:33.217002 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:33.216979    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 18:50:33.224999 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:33.224978    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:50:33.745889 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:33.745858    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 18:50:33.791691 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:33.791653    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f46cb85bc-gj9bv"]
Sep 16 18:50:58.816219 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:58.816160    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f46cb85bc-gj9bv" podUID="eb016dc4-87d2-4ddf-86a9-0b1a90667e50" containerName="console" containerID="cri-o://ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1" gracePeriod=15
Sep 16 18:50:59.052398 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.052371    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:50:59.124856 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.124831    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-oauth-config\") pod \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") "
Sep 16 18:50:59.124993 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.124870    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-config\") pod \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") "
Sep 16 18:50:59.125096 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.125068    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-trusted-ca-bundle\") pod \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") "
Sep 16 18:50:59.125170 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.125154    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-service-ca\") pod \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") "
Sep 16 18:50:59.125231 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.125185    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-serving-cert\") pod \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") "
Sep 16 18:50:59.125231 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.125219    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-oauth-serving-cert\") pod \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") "
Sep 16 18:50:59.125338 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.125296    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knnwc\" (UniqueName: \"kubernetes.io/projected/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-kube-api-access-knnwc\") pod \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\" (UID: \"eb016dc4-87d2-4ddf-86a9-0b1a90667e50\") "
Sep 16 18:50:59.125395 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.125360    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-config" (OuterVolumeSpecName: "console-config") pod "eb016dc4-87d2-4ddf-86a9-0b1a90667e50" (UID: "eb016dc4-87d2-4ddf-86a9-0b1a90667e50"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:59.125597 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.125565    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:50:59.125728 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.125499    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "eb016dc4-87d2-4ddf-86a9-0b1a90667e50" (UID: "eb016dc4-87d2-4ddf-86a9-0b1a90667e50"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:59.125728 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.125627    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-service-ca" (OuterVolumeSpecName: "service-ca") pod "eb016dc4-87d2-4ddf-86a9-0b1a90667e50" (UID: "eb016dc4-87d2-4ddf-86a9-0b1a90667e50"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:59.125842 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.125787    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eb016dc4-87d2-4ddf-86a9-0b1a90667e50" (UID: "eb016dc4-87d2-4ddf-86a9-0b1a90667e50"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:59.127485 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.127458    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eb016dc4-87d2-4ddf-86a9-0b1a90667e50" (UID: "eb016dc4-87d2-4ddf-86a9-0b1a90667e50"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:59.127794 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.127766    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eb016dc4-87d2-4ddf-86a9-0b1a90667e50" (UID: "eb016dc4-87d2-4ddf-86a9-0b1a90667e50"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:59.127857 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.127778    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-kube-api-access-knnwc" (OuterVolumeSpecName: "kube-api-access-knnwc") pod "eb016dc4-87d2-4ddf-86a9-0b1a90667e50" (UID: "eb016dc4-87d2-4ddf-86a9-0b1a90667e50"). InnerVolumeSpecName "kube-api-access-knnwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:59.226385 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.226350    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-trusted-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:50:59.226385 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.226380    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:50:59.226558 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.226393    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:50:59.226558 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.226405    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-oauth-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:50:59.226558 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.226417    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knnwc\" (UniqueName: \"kubernetes.io/projected/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-kube-api-access-knnwc\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:50:59.226558 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.226429    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb016dc4-87d2-4ddf-86a9-0b1a90667e50-console-oauth-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:50:59.847040 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.847003    2567 generic.go:356] "Generic (PLEG): container finished" podID="eb016dc4-87d2-4ddf-86a9-0b1a90667e50" containerID="ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1" exitCode=0
Sep 16 18:50:59.847525 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.847064    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f46cb85bc-gj9bv"
Sep 16 18:50:59.847525 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.847090    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f46cb85bc-gj9bv" event={"ID":"eb016dc4-87d2-4ddf-86a9-0b1a90667e50","Type":"ContainerDied","Data":"ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1"}
Sep 16 18:50:59.847525 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.847222    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f46cb85bc-gj9bv" event={"ID":"eb016dc4-87d2-4ddf-86a9-0b1a90667e50","Type":"ContainerDied","Data":"5a1e75b45a828a2788a66a11fc14bd8ed27462dea1057266edc492f2d9a0373d"}
Sep 16 18:50:59.847525 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.847252    2567 scope.go:117] "RemoveContainer" containerID="ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1"
Sep 16 18:50:59.855629 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.855613    2567 scope.go:117] "RemoveContainer" containerID="ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1"
Sep 16 18:50:59.855897 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:50:59.855879    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1\": container with ID starting with ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1 not found: ID does not exist" containerID="ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1"
Sep 16 18:50:59.855950 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.855909    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1"} err="failed to get container status \"ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1\": rpc error: code = NotFound desc = could not find container \"ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1\": container with ID starting with ead89de9f7e5c8a3553ac75a228646cbe8b79a93cfa259fbe167277981e3cac1 not found: ID does not exist"
Sep 16 18:50:59.870414 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.870385    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f46cb85bc-gj9bv"]
Sep 16 18:50:59.875552 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:50:59.875533    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f46cb85bc-gj9bv"]
Sep 16 18:51:01.333450 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:51:01.333419    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb016dc4-87d2-4ddf-86a9-0b1a90667e50" path="/var/lib/kubelet/pods/eb016dc4-87d2-4ddf-86a9-0b1a90667e50/volumes"
Sep 16 18:53:21.191641 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.191603    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-rtp9z"]
Sep 16 18:53:21.192103 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.191951    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb016dc4-87d2-4ddf-86a9-0b1a90667e50" containerName="console"
Sep 16 18:53:21.192103 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.191964    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb016dc4-87d2-4ddf-86a9-0b1a90667e50" containerName="console"
Sep 16 18:53:21.192103 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.192038    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb016dc4-87d2-4ddf-86a9-0b1a90667e50" containerName="console"
Sep 16 18:53:21.195007 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.194992    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rtp9z"
Sep 16 18:53:21.198440 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.198414    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ffpfr\""
Sep 16 18:53:21.198570 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.198471    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:53:21.199334 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.199316    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:53:21.199430 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.199348    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:53:21.209669 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.209649    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rtp9z"]
Sep 16 18:53:21.225376 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.225345    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npscv\" (UniqueName: \"kubernetes.io/projected/3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2-kube-api-access-npscv\") pod \"seaweedfs-64568bcd49-rtp9z\" (UID: \"3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2\") " pod="kserve/seaweedfs-64568bcd49-rtp9z"
Sep 16 18:53:21.225518 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.225382    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2-data\") pod \"seaweedfs-64568bcd49-rtp9z\" (UID: \"3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2\") " pod="kserve/seaweedfs-64568bcd49-rtp9z"
Sep 16 18:53:21.326952 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.326917    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npscv\" (UniqueName: \"kubernetes.io/projected/3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2-kube-api-access-npscv\") pod \"seaweedfs-64568bcd49-rtp9z\" (UID: \"3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2\") " pod="kserve/seaweedfs-64568bcd49-rtp9z"
Sep 16 18:53:21.327086 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.326963    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2-data\") pod \"seaweedfs-64568bcd49-rtp9z\" (UID: \"3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2\") " pod="kserve/seaweedfs-64568bcd49-rtp9z"
Sep 16 18:53:21.327362 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.327343    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2-data\") pod \"seaweedfs-64568bcd49-rtp9z\" (UID: \"3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2\") " pod="kserve/seaweedfs-64568bcd49-rtp9z"
Sep 16 18:53:21.335748 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.335719    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npscv\" (UniqueName: \"kubernetes.io/projected/3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2-kube-api-access-npscv\") pod \"seaweedfs-64568bcd49-rtp9z\" (UID: \"3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2\") " pod="kserve/seaweedfs-64568bcd49-rtp9z"
Sep 16 18:53:21.505613 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.505535    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rtp9z"
Sep 16 18:53:21.775076 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.775050    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rtp9z"]
Sep 16 18:53:21.777750 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:53:21.777719    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbcb5f1_2da0_42c5_97c8_ab3e8f7766f2.slice/crio-d36b5eb0b1d2a6551fd35b3e33c026bd535e23d6d88fc60ca6651ff1f6f2fc17 WatchSource:0}: Error finding container d36b5eb0b1d2a6551fd35b3e33c026bd535e23d6d88fc60ca6651ff1f6f2fc17: Status 404 returned error can't find the container with id d36b5eb0b1d2a6551fd35b3e33c026bd535e23d6d88fc60ca6651ff1f6f2fc17
Sep 16 18:53:21.779181 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:21.779160    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:53:22.431516 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:22.431477    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rtp9z" event={"ID":"3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2","Type":"ContainerStarted","Data":"d36b5eb0b1d2a6551fd35b3e33c026bd535e23d6d88fc60ca6651ff1f6f2fc17"}
Sep 16 18:53:25.444580 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:25.444540    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rtp9z" event={"ID":"3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2","Type":"ContainerStarted","Data":"27fded86c5e4dda83c00f56d8c1d61ae1e7a0163944132a3ec46ac56002da273"}
Sep 16 18:53:25.444926 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:25.444673    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-rtp9z"
Sep 16 18:53:25.461991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:25.461936    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-rtp9z" podStartSLOduration=1.40311596 podStartE2EDuration="4.46189142s" totalImagesPullingTime="3.05877546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:25.461462146 +0000 UTC m=+472.649908829" watchObservedRunningTime="2026-09-16 18:53:25.46189142 +0000 UTC m=+472.650338102"
Sep 16 18:53:31.451409 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:53:31.451379    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-rtp9z"
Sep 16 18:54:55.653790 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.653714    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-vstz2"]
Sep 16 18:54:55.656801 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.656785    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vstz2"
Sep 16 18:54:55.659610 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.659587    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:54:55.665671 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.665649    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vstz2"]
Sep 16 18:54:55.810734 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.810695    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7l7r\" (UniqueName: \"kubernetes.io/projected/fa85cfaf-4aa1-4861-afda-64c62b1a566a-kube-api-access-c7l7r\") pod \"s3-init-vstz2\" (UID: \"fa85cfaf-4aa1-4861-afda-64c62b1a566a\") " pod="kserve/s3-init-vstz2"
Sep 16 18:54:55.810886 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.810766    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa85cfaf-4aa1-4861-afda-64c62b1a566a-model-cache\") pod \"s3-init-vstz2\" (UID: \"fa85cfaf-4aa1-4861-afda-64c62b1a566a\") " pod="kserve/s3-init-vstz2"
Sep 16 18:54:55.912019 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.911943    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa85cfaf-4aa1-4861-afda-64c62b1a566a-model-cache\") pod \"s3-init-vstz2\" (UID: \"fa85cfaf-4aa1-4861-afda-64c62b1a566a\") " pod="kserve/s3-init-vstz2"
Sep 16 18:54:55.912180 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.912022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7l7r\" (UniqueName: \"kubernetes.io/projected/fa85cfaf-4aa1-4861-afda-64c62b1a566a-kube-api-access-c7l7r\") pod \"s3-init-vstz2\" (UID: \"fa85cfaf-4aa1-4861-afda-64c62b1a566a\") " pod="kserve/s3-init-vstz2"
Sep 16 18:54:55.912410 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.912391    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa85cfaf-4aa1-4861-afda-64c62b1a566a-model-cache\") pod \"s3-init-vstz2\" (UID: \"fa85cfaf-4aa1-4861-afda-64c62b1a566a\") " pod="kserve/s3-init-vstz2"
Sep 16 18:54:55.921249 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.921229    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7l7r\" (UniqueName: \"kubernetes.io/projected/fa85cfaf-4aa1-4861-afda-64c62b1a566a-kube-api-access-c7l7r\") pod \"s3-init-vstz2\" (UID: \"fa85cfaf-4aa1-4861-afda-64c62b1a566a\") " pod="kserve/s3-init-vstz2"
Sep 16 18:54:55.976167 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:55.976135    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vstz2"
Sep 16 18:54:56.246242 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:56.246164    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vstz2"]
Sep 16 18:54:56.249538 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:54:56.249509    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa85cfaf_4aa1_4861_afda_64c62b1a566a.slice/crio-22407f80e1879bad060fca841e1e8d41a76dafb5b9491077cd145dd6b94bafbf WatchSource:0}: Error finding container 22407f80e1879bad060fca841e1e8d41a76dafb5b9491077cd145dd6b94bafbf: Status 404 returned error can't find the container with id 22407f80e1879bad060fca841e1e8d41a76dafb5b9491077cd145dd6b94bafbf
Sep 16 18:54:56.829264 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:54:56.829224    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vstz2" event={"ID":"fa85cfaf-4aa1-4861-afda-64c62b1a566a","Type":"ContainerStarted","Data":"22407f80e1879bad060fca841e1e8d41a76dafb5b9491077cd145dd6b94bafbf"}
Sep 16 18:55:00.851231 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:55:00.851190    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vstz2" event={"ID":"fa85cfaf-4aa1-4861-afda-64c62b1a566a","Type":"ContainerStarted","Data":"5a903887fe9346ad0d5a5fc39831008940f4bcebb13c14b3fb8e24a02bab8d8d"}
Sep 16 18:55:02.862535 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:55:02.862502    2567 generic.go:356] "Generic (PLEG): container finished" podID="fa85cfaf-4aa1-4861-afda-64c62b1a566a" containerID="5a903887fe9346ad0d5a5fc39831008940f4bcebb13c14b3fb8e24a02bab8d8d" exitCode=0
Sep 16 18:55:02.862983 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:55:02.862578    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vstz2" event={"ID":"fa85cfaf-4aa1-4861-afda-64c62b1a566a","Type":"ContainerDied","Data":"5a903887fe9346ad0d5a5fc39831008940f4bcebb13c14b3fb8e24a02bab8d8d"}
Sep 16 18:55:07.891743 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:55:07.891708    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vstz2" event={"ID":"fa85cfaf-4aa1-4861-afda-64c62b1a566a","Type":"ContainerStarted","Data":"f6bc6d14107cccde8e9c5d9b055ce5dee724a8333bbef11b684be7c1def08b05"}
Sep 16 18:55:07.911578 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:55:07.911533    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-vstz2" podStartSLOduration=2.775879377 podStartE2EDuration="12.911518033s" totalImagesPullingTime="8.618833261s" totalInitContainerRuntime="1.516805395s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:07.910615663 +0000 UTC m=+575.099062347" watchObservedRunningTime="2026-09-16 18:55:07.911518033 +0000 UTC m=+575.099964714"
Sep 16 18:55:33.246938 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:55:33.246910    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 18:55:33.247474 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:55:33.247043    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 18:56:13.160570 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:13.160536    2567 generic.go:356] "Generic (PLEG): container finished" podID="fa85cfaf-4aa1-4861-afda-64c62b1a566a" containerID="f6bc6d14107cccde8e9c5d9b055ce5dee724a8333bbef11b684be7c1def08b05" exitCode=0
Sep 16 18:56:13.160967 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:13.160598    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vstz2" event={"ID":"fa85cfaf-4aa1-4861-afda-64c62b1a566a","Type":"ContainerDied","Data":"f6bc6d14107cccde8e9c5d9b055ce5dee724a8333bbef11b684be7c1def08b05"}
Sep 16 18:56:14.283165 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:14.283098    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vstz2"
Sep 16 18:56:14.390737 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:14.390697    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa85cfaf-4aa1-4861-afda-64c62b1a566a-model-cache\") pod \"fa85cfaf-4aa1-4861-afda-64c62b1a566a\" (UID: \"fa85cfaf-4aa1-4861-afda-64c62b1a566a\") "
Sep 16 18:56:14.390923 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:14.390818    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7l7r\" (UniqueName: \"kubernetes.io/projected/fa85cfaf-4aa1-4861-afda-64c62b1a566a-kube-api-access-c7l7r\") pod \"fa85cfaf-4aa1-4861-afda-64c62b1a566a\" (UID: \"fa85cfaf-4aa1-4861-afda-64c62b1a566a\") "
Sep 16 18:56:14.393209 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:14.393180    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa85cfaf-4aa1-4861-afda-64c62b1a566a-kube-api-access-c7l7r" (OuterVolumeSpecName: "kube-api-access-c7l7r") pod "fa85cfaf-4aa1-4861-afda-64c62b1a566a" (UID: "fa85cfaf-4aa1-4861-afda-64c62b1a566a"). InnerVolumeSpecName "kube-api-access-c7l7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:14.408975 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:14.408950    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa85cfaf-4aa1-4861-afda-64c62b1a566a-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa85cfaf-4aa1-4861-afda-64c62b1a566a" (UID: "fa85cfaf-4aa1-4861-afda-64c62b1a566a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:14.492159 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:14.492079    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7l7r\" (UniqueName: \"kubernetes.io/projected/fa85cfaf-4aa1-4861-afda-64c62b1a566a-kube-api-access-c7l7r\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:56:14.492159 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:14.492103    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa85cfaf-4aa1-4861-afda-64c62b1a566a-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:56:15.172121 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:15.172093    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vstz2"
Sep 16 18:56:15.172297 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:15.172090    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vstz2" event={"ID":"fa85cfaf-4aa1-4861-afda-64c62b1a566a","Type":"ContainerDied","Data":"22407f80e1879bad060fca841e1e8d41a76dafb5b9491077cd145dd6b94bafbf"}
Sep 16 18:56:15.172297 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:15.172225    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22407f80e1879bad060fca841e1e8d41a76dafb5b9491077cd145dd6b94bafbf"
Sep 16 18:56:27.721147 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.721099    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"]
Sep 16 18:56:27.721629 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.721615    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa85cfaf-4aa1-4861-afda-64c62b1a566a" containerName="download-hf-model"
Sep 16 18:56:27.721672 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.721632    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa85cfaf-4aa1-4861-afda-64c62b1a566a" containerName="download-hf-model"
Sep 16 18:56:27.721672 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.721648    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa85cfaf-4aa1-4861-afda-64c62b1a566a" containerName="s3-init"
Sep 16 18:56:27.721672 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.721657    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa85cfaf-4aa1-4861-afda-64c62b1a566a" containerName="s3-init"
Sep 16 18:56:27.721763 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.721727    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa85cfaf-4aa1-4861-afda-64c62b1a566a" containerName="s3-init"
Sep 16 18:56:27.724220 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.724196    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:27.727282 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.727255    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:56:27.727402 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.727256    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 18:56:27.727402 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.727358    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:56:27.727632 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.727613    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 18:56:27.727699 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.727640    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\""
Sep 16 18:56:27.727753 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.727704    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:56:27.728655 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.728641    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:56:27.735448 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.735425    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"]
Sep 16 18:56:27.861708 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.861676    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"]
Sep 16 18:56:27.864171 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.864154    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:27.882272 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.882251    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:56:27.899481 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.899454    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"]
Sep 16 18:56:27.902929 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.902902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d397a5ad-2723-4616-8d4e-e404da2a4c0d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:27.903042 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.903004    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pdz\" (UniqueName: \"kubernetes.io/projected/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-api-access-62pdz\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:27.903101 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.903049    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:27.903189 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.903173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:27.903274 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:27.903249    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.004743 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.004656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d397a5ad-2723-4616-8d4e-e404da2a4c0d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.004743 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.004704    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.004743 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.004728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pdz\" (UniqueName: \"kubernetes.io/projected/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-api-access-62pdz\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.005197 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.004775    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.005197 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.004805    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.005197 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.004861    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.005197 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.004914    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkk5\" (UniqueName: \"kubernetes.io/projected/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-api-access-fqkk5\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.005197 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.004942    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.005197 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.004986    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.005197 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.005020    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.005455 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.005290    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.005559 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.005541    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.005597 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.005581    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.007230 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.007208    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d397a5ad-2723-4616-8d4e-e404da2a4c0d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.016727 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.016697    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pdz\" (UniqueName: \"kubernetes.io/projected/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-api-access-62pdz\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.036284 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.036259    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:28.105805 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.105775    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.105979 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.105842    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.105979 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.105879    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkk5\" (UniqueName: \"kubernetes.io/projected/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-api-access-fqkk5\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.105979 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.105928    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.105979 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.105976    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.106331 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.106311    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.106562 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.106531    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.106673 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.106649    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.108617 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.108580    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.115594 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.115574    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkk5\" (UniqueName: \"kubernetes.io/projected/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-api-access-fqkk5\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-b75qt\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.173785 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.173755    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:28.312584 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.312560    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"]
Sep 16 18:56:28.314516 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:56:28.314485    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd397a5ad_2723_4616_8d4e_e404da2a4c0d.slice/crio-6ae8d7ec07628cbef7df9df6ad6cfce59bf0bd6cdd69edf7f834c45b5be88ecb WatchSource:0}: Error finding container 6ae8d7ec07628cbef7df9df6ad6cfce59bf0bd6cdd69edf7f834c45b5be88ecb: Status 404 returned error can't find the container with id 6ae8d7ec07628cbef7df9df6ad6cfce59bf0bd6cdd69edf7f834c45b5be88ecb
Sep 16 18:56:28.448448 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:28.448424    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"]
Sep 16 18:56:28.450314 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:56:28.450272    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e53ff5_93d6_4cbe_9c3f_2cc3b32472cb.slice/crio-7d600adf3e777cec7532df032f97ba15320bf05c07b0555e7b08925c23d5a2ca WatchSource:0}: Error finding container 7d600adf3e777cec7532df032f97ba15320bf05c07b0555e7b08925c23d5a2ca: Status 404 returned error can't find the container with id 7d600adf3e777cec7532df032f97ba15320bf05c07b0555e7b08925c23d5a2ca
Sep 16 18:56:29.239031 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:29.238995    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" event={"ID":"d397a5ad-2723-4616-8d4e-e404da2a4c0d","Type":"ContainerStarted","Data":"bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a"}
Sep 16 18:56:29.239487 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:29.239074    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" event={"ID":"d397a5ad-2723-4616-8d4e-e404da2a4c0d","Type":"ContainerStarted","Data":"6ae8d7ec07628cbef7df9df6ad6cfce59bf0bd6cdd69edf7f834c45b5be88ecb"}
Sep 16 18:56:29.240550 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:29.240528    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" event={"ID":"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb","Type":"ContainerStarted","Data":"30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766"}
Sep 16 18:56:29.240655 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:29.240576    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" event={"ID":"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb","Type":"ContainerStarted","Data":"7d600adf3e777cec7532df032f97ba15320bf05c07b0555e7b08925c23d5a2ca"}
Sep 16 18:56:32.256056 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:32.255990    2567 generic.go:356] "Generic (PLEG): container finished" podID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerID="bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a" exitCode=0
Sep 16 18:56:32.256457 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:32.256068    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" event={"ID":"d397a5ad-2723-4616-8d4e-e404da2a4c0d","Type":"ContainerDied","Data":"bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a"}
Sep 16 18:56:32.257753 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:32.257730    2567 generic.go:356] "Generic (PLEG): container finished" podID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerID="30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766" exitCode=0
Sep 16 18:56:32.257916 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:32.257797    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" event={"ID":"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb","Type":"ContainerDied","Data":"30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766"}
Sep 16 18:56:49.417642 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:49.417541    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" event={"ID":"d397a5ad-2723-4616-8d4e-e404da2a4c0d","Type":"ContainerStarted","Data":"946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061"}
Sep 16 18:56:49.420914 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:49.420835    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" event={"ID":"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb","Type":"ContainerStarted","Data":"84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789"}
Sep 16 18:56:52.437486 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:52.437444    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" event={"ID":"d397a5ad-2723-4616-8d4e-e404da2a4c0d","Type":"ContainerStarted","Data":"a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745"}
Sep 16 18:56:52.437932 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:52.437516    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:52.437932 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:52.437557    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:52.439245 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:52.439220    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:56:52.439781 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:52.439759    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" event={"ID":"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb","Type":"ContainerStarted","Data":"7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c"}
Sep 16 18:56:52.439901 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:52.439885    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:52.478760 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:52.478700    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podStartSLOduration=2.691472531 podStartE2EDuration="25.478683224s" totalImagesPullingTime="19.196069602s" totalInitContainerRuntime="3.591141091s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:52.476568469 +0000 UTC m=+679.665015150" watchObservedRunningTime="2026-09-16 18:56:52.478683224 +0000 UTC m=+679.667129905"
Sep 16 18:56:52.507831 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:52.507785    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podStartSLOduration=2.483869191 podStartE2EDuration="25.507768632s" totalImagesPullingTime="19.330602746s" totalInitContainerRuntime="3.693296695s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:52.506778197 +0000 UTC m=+679.695224879" watchObservedRunningTime="2026-09-16 18:56:52.507768632 +0000 UTC m=+679.696215316"
Sep 16 18:56:53.444657 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:53.444629    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:53.445359 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:53.444617    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:56:53.445768 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:53.445745    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:56:54.449236 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:54.449195    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:56:58.449514 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:58.449483    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:56:58.450052 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:58.450022    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:56:59.456043 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:59.456016    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:56:59.456663 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:56:59.456634    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:57:08.449991 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:08.449956    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:57:09.457127 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:09.457068    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:57:18.450434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:18.450388    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:57:19.457713 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:19.457667    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:57:28.450088 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:28.450010    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:57:29.457494 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:29.457452    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:57:38.450584 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:38.450541    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:57:39.456746 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:39.456703    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:57:48.450407 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:48.450362    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:57:49.456989 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:49.456940    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:57:58.450088 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:58.450043    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:57:59.457043 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:57:59.457001    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:58:08.451307 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:08.451270    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:58:08.596757 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:08.596719    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"]
Sep 16 18:58:08.597003 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:08.596965    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container" containerID="cri-o://84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789" gracePeriod=30
Sep 16 18:58:08.597174 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:08.597015    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kube-rbac-proxy" containerID="cri-o://7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c" gracePeriod=30
Sep 16 18:58:08.597870 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:08.597851    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:58:08.793055 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:08.793023    2567 generic.go:356] "Generic (PLEG): container finished" podID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerID="7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c" exitCode=2
Sep 16 18:58:08.793238 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:08.793097    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" event={"ID":"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb","Type":"ContainerDied","Data":"7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c"}
Sep 16 18:58:09.450188 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:09.450148    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 18:58:13.138697 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.138673    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:58:13.173819 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.173791    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-cabundle-cert\") pod \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") "
Sep 16 18:58:13.173980 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.173864    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kserve-provision-location\") pod \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") "
Sep 16 18:58:13.173980 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.173894    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqkk5\" (UniqueName: \"kubernetes.io/projected/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-api-access-fqkk5\") pod \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") "
Sep 16 18:58:13.173980 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.173916    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-rbac-proxy-sar-config\") pod \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") "
Sep 16 18:58:13.173980 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.173965    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-proxy-tls\") pod \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\" (UID: \"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb\") "
Sep 16 18:58:13.174300 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.174193    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" (UID: "f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:13.174300 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.174234    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" (UID: "f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:13.174502 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.174478    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" (UID: "f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:13.176349 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.176320    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" (UID: "f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:13.176478 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.176412    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-api-access-fqkk5" (OuterVolumeSpecName: "kube-api-access-fqkk5") pod "f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" (UID: "f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb"). InnerVolumeSpecName "kube-api-access-fqkk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:13.275308 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.275229    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:58:13.275308 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.275260    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:58:13.275308 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.275270    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqkk5\" (UniqueName: \"kubernetes.io/projected/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-api-access-fqkk5\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:58:13.275308 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.275282    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:58:13.275308 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.275290    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:58:13.818449 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.818409    2567 generic.go:356] "Generic (PLEG): container finished" podID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerID="84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789" exitCode=0
Sep 16 18:58:13.818617 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.818476    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"
Sep 16 18:58:13.818617 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.818479    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" event={"ID":"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb","Type":"ContainerDied","Data":"84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789"}
Sep 16 18:58:13.818617 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.818540    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt" event={"ID":"f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb","Type":"ContainerDied","Data":"7d600adf3e777cec7532df032f97ba15320bf05c07b0555e7b08925c23d5a2ca"}
Sep 16 18:58:13.818617 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.818566    2567 scope.go:117] "RemoveContainer" containerID="7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c"
Sep 16 18:58:13.827075 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.827056    2567 scope.go:117] "RemoveContainer" containerID="84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789"
Sep 16 18:58:13.836647 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.836625    2567 scope.go:117] "RemoveContainer" containerID="30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766"
Sep 16 18:58:13.842443 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.842419    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"]
Sep 16 18:58:13.844851 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.844831    2567 scope.go:117] "RemoveContainer" containerID="7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c"
Sep 16 18:58:13.845176 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:58:13.845149    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c\": container with ID starting with 7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c not found: ID does not exist" containerID="7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c"
Sep 16 18:58:13.845311 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.845188    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c"} err="failed to get container status \"7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c\": rpc error: code = NotFound desc = could not find container \"7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c\": container with ID starting with 7f543cad84c6a4663b1c444ef2bb8eb20daa1d3084559a7e39e4df93546e1e6c not found: ID does not exist"
Sep 16 18:58:13.845311 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.845216    2567 scope.go:117] "RemoveContainer" containerID="84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789"
Sep 16 18:58:13.845913 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:58:13.845890    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789\": container with ID starting with 84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789 not found: ID does not exist" containerID="84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789"
Sep 16 18:58:13.846011 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.845917    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789"} err="failed to get container status \"84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789\": rpc error: code = NotFound desc = could not find container \"84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789\": container with ID starting with 84edd868d2aaebb5a3b138260e273772d9c4e73d21aff8f3ccfc60c46c9ed789 not found: ID does not exist"
Sep 16 18:58:13.846011 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.845933    2567 scope.go:117] "RemoveContainer" containerID="30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766"
Sep 16 18:58:13.846280 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:58:13.846257    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766\": container with ID starting with 30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766 not found: ID does not exist" containerID="30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766"
Sep 16 18:58:13.846359 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.846289    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766"} err="failed to get container status \"30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766\": rpc error: code = NotFound desc = could not find container \"30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766\": container with ID starting with 30f7a17c54b9f36aa4ba5d1443e6070f79f57f04e96ededc31df0a6623e36766 not found: ID does not exist"
Sep 16 18:58:13.847467 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:13.847448    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-b75qt"]
Sep 16 18:58:15.333448 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:15.333413    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" path="/var/lib/kubelet/pods/f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb/volumes"
Sep 16 18:58:17.614406 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.614317    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"]
Sep 16 18:58:17.614845 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.614594    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" containerID="cri-o://946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061" gracePeriod=30
Sep 16 18:58:17.614845 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.614672    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kube-rbac-proxy" containerID="cri-o://a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745" gracePeriod=30
Sep 16 18:58:17.714374 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.714342    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"]
Sep 16 18:58:17.714687 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.714676    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container"
Sep 16 18:58:17.714732 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.714689    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container"
Sep 16 18:58:17.714732 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.714700    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kube-rbac-proxy"
Sep 16 18:58:17.714732 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.714706    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kube-rbac-proxy"
Sep 16 18:58:17.714732 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.714719    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="storage-initializer"
Sep 16 18:58:17.714732 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.714724    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="storage-initializer"
Sep 16 18:58:17.714908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.714787    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kserve-container"
Sep 16 18:58:17.714908 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.714795    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e53ff5-93d6-4cbe-9c3f-2cc3b32472cb" containerName="kube-rbac-proxy"
Sep 16 18:58:17.719501 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.719479    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.722380 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.722359    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:58:17.731341 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.731321    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"]
Sep 16 18:58:17.813304 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.813266    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.813468 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.813349    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f732d8-64d9-4693-b946-f9d08bc9023a-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.813468 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.813380    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.813599 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.813461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h99xx\" (UniqueName: \"kubernetes.io/projected/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-api-access-h99xx\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.813599 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.813539    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f732d8-64d9-4693-b946-f9d08bc9023a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.840146 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.840104    2567 generic.go:356] "Generic (PLEG): container finished" podID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerID="a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745" exitCode=2
Sep 16 18:58:17.840274 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.840165    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" event={"ID":"d397a5ad-2723-4616-8d4e-e404da2a4c0d","Type":"ContainerDied","Data":"a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745"}
Sep 16 18:58:17.914661 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.914626    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f732d8-64d9-4693-b946-f9d08bc9023a-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.914814 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.914669    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.914814 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.914726    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h99xx\" (UniqueName: \"kubernetes.io/projected/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-api-access-h99xx\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.914814 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.914792    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f732d8-64d9-4693-b946-f9d08bc9023a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.914951 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.914828    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.915277 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.915255    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f732d8-64d9-4693-b946-f9d08bc9023a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.915433 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.915414    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.915486 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.915471    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.917148 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.917132    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f732d8-64d9-4693-b946-f9d08bc9023a-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:17.924417 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:17.924396    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h99xx\" (UniqueName: \"kubernetes.io/projected/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-api-access-h99xx\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-n29rd\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:18.030405 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:18.030358    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:18.310730 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:18.310695    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"]
Sep 16 18:58:18.313001 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:58:18.312970    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f732d8_64d9_4693_b946_f9d08bc9023a.slice/crio-f1df0cc4c559e71d1015618b986fd94a84941f44a235dafb9a87ef0118b94c46 WatchSource:0}: Error finding container f1df0cc4c559e71d1015618b986fd94a84941f44a235dafb9a87ef0118b94c46: Status 404 returned error can't find the container with id f1df0cc4c559e71d1015618b986fd94a84941f44a235dafb9a87ef0118b94c46
Sep 16 18:58:18.445209 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:18.445167    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 18:58:18.450717 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:18.450684    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:58:18.847129 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:18.847087    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" event={"ID":"e4f732d8-64d9-4693-b946-f9d08bc9023a","Type":"ContainerStarted","Data":"c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53"}
Sep 16 18:58:18.847551 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:18.847158    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" event={"ID":"e4f732d8-64d9-4693-b946-f9d08bc9023a","Type":"ContainerStarted","Data":"f1df0cc4c559e71d1015618b986fd94a84941f44a235dafb9a87ef0118b94c46"}
Sep 16 18:58:22.097306 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.097285    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:58:22.149966 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.149937    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kserve-provision-location\") pod \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") "
Sep 16 18:58:22.150125 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.149987    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-cabundle-cert\") pod \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") "
Sep 16 18:58:22.150125 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.150072    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62pdz\" (UniqueName: \"kubernetes.io/projected/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-api-access-62pdz\") pod \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") "
Sep 16 18:58:22.150289 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.150141    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d397a5ad-2723-4616-8d4e-e404da2a4c0d-proxy-tls\") pod \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") "
Sep 16 18:58:22.150289 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.150169    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-rbac-proxy-sar-config\") pod \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\" (UID: \"d397a5ad-2723-4616-8d4e-e404da2a4c0d\") "
Sep 16 18:58:22.150405 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.150314    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d397a5ad-2723-4616-8d4e-e404da2a4c0d" (UID: "d397a5ad-2723-4616-8d4e-e404da2a4c0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:22.150463 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.150419    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d397a5ad-2723-4616-8d4e-e404da2a4c0d" (UID: "d397a5ad-2723-4616-8d4e-e404da2a4c0d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:22.150568 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.150464    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:58:22.150684 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.150566    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d397a5ad-2723-4616-8d4e-e404da2a4c0d" (UID: "d397a5ad-2723-4616-8d4e-e404da2a4c0d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:22.152171 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.152144    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-api-access-62pdz" (OuterVolumeSpecName: "kube-api-access-62pdz") pod "d397a5ad-2723-4616-8d4e-e404da2a4c0d" (UID: "d397a5ad-2723-4616-8d4e-e404da2a4c0d"). InnerVolumeSpecName "kube-api-access-62pdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:22.152246 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.152172    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d397a5ad-2723-4616-8d4e-e404da2a4c0d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d397a5ad-2723-4616-8d4e-e404da2a4c0d" (UID: "d397a5ad-2723-4616-8d4e-e404da2a4c0d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:22.251854 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.251822    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d397a5ad-2723-4616-8d4e-e404da2a4c0d-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:58:22.251854 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.251848    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:58:22.251854 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.251857    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d397a5ad-2723-4616-8d4e-e404da2a4c0d-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:58:22.252044 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.251866    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62pdz\" (UniqueName: \"kubernetes.io/projected/d397a5ad-2723-4616-8d4e-e404da2a4c0d-kube-api-access-62pdz\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:58:22.868832 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.868801    2567 generic.go:356] "Generic (PLEG): container finished" podID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerID="946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061" exitCode=0
Sep 16 18:58:22.869020 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.868857    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"
Sep 16 18:58:22.869020 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.868885    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" event={"ID":"d397a5ad-2723-4616-8d4e-e404da2a4c0d","Type":"ContainerDied","Data":"946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061"}
Sep 16 18:58:22.869020 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.868959    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq" event={"ID":"d397a5ad-2723-4616-8d4e-e404da2a4c0d","Type":"ContainerDied","Data":"6ae8d7ec07628cbef7df9df6ad6cfce59bf0bd6cdd69edf7f834c45b5be88ecb"}
Sep 16 18:58:22.869020 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.868997    2567 scope.go:117] "RemoveContainer" containerID="a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745"
Sep 16 18:58:22.870795 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.870767    2567 generic.go:356] "Generic (PLEG): container finished" podID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerID="c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53" exitCode=0
Sep 16 18:58:22.870893 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.870805    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" event={"ID":"e4f732d8-64d9-4693-b946-f9d08bc9023a","Type":"ContainerDied","Data":"c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53"}
Sep 16 18:58:22.880802 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.880786    2567 scope.go:117] "RemoveContainer" containerID="946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061"
Sep 16 18:58:22.890648 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.890626    2567 scope.go:117] "RemoveContainer" containerID="bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a"
Sep 16 18:58:22.909205 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.909182    2567 scope.go:117] "RemoveContainer" containerID="a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745"
Sep 16 18:58:22.909543 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:58:22.909520    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745\": container with ID starting with a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745 not found: ID does not exist" containerID="a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745"
Sep 16 18:58:22.909634 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.909554    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745"} err="failed to get container status \"a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745\": rpc error: code = NotFound desc = could not find container \"a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745\": container with ID starting with a42a2edf6339da9d24608150e43180f8dfec4a2eca167424369808afb9e14745 not found: ID does not exist"
Sep 16 18:58:22.909634 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.909570    2567 scope.go:117] "RemoveContainer" containerID="946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061"
Sep 16 18:58:22.909880 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:58:22.909862    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061\": container with ID starting with 946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061 not found: ID does not exist" containerID="946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061"
Sep 16 18:58:22.909948 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.909882    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061"} err="failed to get container status \"946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061\": rpc error: code = NotFound desc = could not find container \"946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061\": container with ID starting with 946542e50f49e5c7f78b1dd4cf11a1802f172af58251d64d456904ca5359f061 not found: ID does not exist"
Sep 16 18:58:22.909948 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.909908    2567 scope.go:117] "RemoveContainer" containerID="bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a"
Sep 16 18:58:22.910211 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:58:22.910174    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a\": container with ID starting with bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a not found: ID does not exist" containerID="bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a"
Sep 16 18:58:22.910295 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.910227    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a"} err="failed to get container status \"bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a\": rpc error: code = NotFound desc = could not find container \"bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a\": container with ID starting with bf0a81e2285a4239758f8e352abbe86dd6eb75e8415eae5327d62ca8ce17986a not found: ID does not exist"
Sep 16 18:58:22.949858 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.949833    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"]
Sep 16 18:58:22.958635 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:22.958612    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-pdjlq"]
Sep 16 18:58:23.333940 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:23.333849    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" path="/var/lib/kubelet/pods/d397a5ad-2723-4616-8d4e-e404da2a4c0d/volumes"
Sep 16 18:58:23.878585 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:23.878552    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" event={"ID":"e4f732d8-64d9-4693-b946-f9d08bc9023a","Type":"ContainerStarted","Data":"d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9"}
Sep 16 18:58:23.878767 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:23.878610    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" event={"ID":"e4f732d8-64d9-4693-b946-f9d08bc9023a","Type":"ContainerStarted","Data":"b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f"}
Sep 16 18:58:23.878852 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:23.878794    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:23.907484 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:23.907431    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podStartSLOduration=3.264706855 podStartE2EDuration="6.907417331s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.642710476s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:23.902577112 +0000 UTC m=+771.091023793" watchObservedRunningTime="2026-09-16 18:58:23.907417331 +0000 UTC m=+771.095864059"
Sep 16 18:58:24.883788 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:24.883758    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:24.884987 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:24.884960    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:58:25.888625 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:25.888582    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:58:30.893327 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:30.893298    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:58:30.893786 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:30.893760    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:58:40.893850 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:40.893810    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:58:50.894668 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:58:50.894626    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:59:00.894009 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:00.893922    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:59:10.894555 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:10.894512    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:59:20.893687 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:20.893648    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:59:30.893780 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:30.893745    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:59:33.331072 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:33.331039    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:59:43.333582 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:43.333554    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:59:47.676225 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:47.676193    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"]
Sep 16 18:59:47.676684 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:47.676449    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container" containerID="cri-o://b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f" gracePeriod=30
Sep 16 18:59:47.676684 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:47.676467    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kube-rbac-proxy" containerID="cri-o://d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9" gracePeriod=30
Sep 16 18:59:48.275081 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:48.275047    2567 generic.go:356] "Generic (PLEG): container finished" podID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerID="d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9" exitCode=2
Sep 16 18:59:48.275265 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:48.275091    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" event={"ID":"e4f732d8-64d9-4693-b946-f9d08bc9023a","Type":"ContainerDied","Data":"d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9"}
Sep 16 18:59:50.889712 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:50.889671    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 16 18:59:52.015416 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.015387    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:59:52.049539 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.049506    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-cabundle-cert\") pod \"e4f732d8-64d9-4693-b946-f9d08bc9023a\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") "
Sep 16 18:59:52.049704 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.049552    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f732d8-64d9-4693-b946-f9d08bc9023a-proxy-tls\") pod \"e4f732d8-64d9-4693-b946-f9d08bc9023a\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") "
Sep 16 18:59:52.049704 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.049592    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-rbac-proxy-sar-config\") pod \"e4f732d8-64d9-4693-b946-f9d08bc9023a\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") "
Sep 16 18:59:52.049704 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.049619    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h99xx\" (UniqueName: \"kubernetes.io/projected/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-api-access-h99xx\") pod \"e4f732d8-64d9-4693-b946-f9d08bc9023a\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") "
Sep 16 18:59:52.049864 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.049768    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f732d8-64d9-4693-b946-f9d08bc9023a-kserve-provision-location\") pod \"e4f732d8-64d9-4693-b946-f9d08bc9023a\" (UID: \"e4f732d8-64d9-4693-b946-f9d08bc9023a\") "
Sep 16 18:59:52.049999 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.049974    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4f732d8-64d9-4693-b946-f9d08bc9023a" (UID: "e4f732d8-64d9-4693-b946-f9d08bc9023a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:52.050103 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.050057    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4f732d8-64d9-4693-b946-f9d08bc9023a" (UID: "e4f732d8-64d9-4693-b946-f9d08bc9023a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:52.050103 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.050055    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f732d8-64d9-4693-b946-f9d08bc9023a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4f732d8-64d9-4693-b946-f9d08bc9023a" (UID: "e4f732d8-64d9-4693-b946-f9d08bc9023a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:52.050245 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.050106    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:59:52.051793 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.051771    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f732d8-64d9-4693-b946-f9d08bc9023a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4f732d8-64d9-4693-b946-f9d08bc9023a" (UID: "e4f732d8-64d9-4693-b946-f9d08bc9023a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:52.052100 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.052081    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-api-access-h99xx" (OuterVolumeSpecName: "kube-api-access-h99xx") pod "e4f732d8-64d9-4693-b946-f9d08bc9023a" (UID: "e4f732d8-64d9-4693-b946-f9d08bc9023a"). InnerVolumeSpecName "kube-api-access-h99xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:52.150888 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.150859    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f732d8-64d9-4693-b946-f9d08bc9023a-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:59:52.150888 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.150886    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:59:52.151057 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.150895    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h99xx\" (UniqueName: \"kubernetes.io/projected/e4f732d8-64d9-4693-b946-f9d08bc9023a-kube-api-access-h99xx\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:59:52.151057 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.150906    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f732d8-64d9-4693-b946-f9d08bc9023a-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 18:59:52.296329 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.296297    2567 generic.go:356] "Generic (PLEG): container finished" podID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerID="b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f" exitCode=0
Sep 16 18:59:52.296505 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.296344    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" event={"ID":"e4f732d8-64d9-4693-b946-f9d08bc9023a","Type":"ContainerDied","Data":"b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f"}
Sep 16 18:59:52.296505 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.296356    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"
Sep 16 18:59:52.296505 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.296402    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd" event={"ID":"e4f732d8-64d9-4693-b946-f9d08bc9023a","Type":"ContainerDied","Data":"f1df0cc4c559e71d1015618b986fd94a84941f44a235dafb9a87ef0118b94c46"}
Sep 16 18:59:52.296505 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.296439    2567 scope.go:117] "RemoveContainer" containerID="d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9"
Sep 16 18:59:52.305870 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.305854    2567 scope.go:117] "RemoveContainer" containerID="b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f"
Sep 16 18:59:52.313556 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.313535    2567 scope.go:117] "RemoveContainer" containerID="c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53"
Sep 16 18:59:52.319696 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.319674    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"]
Sep 16 18:59:52.321066 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.321051    2567 scope.go:117] "RemoveContainer" containerID="d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9"
Sep 16 18:59:52.321336 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:59:52.321317    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9\": container with ID starting with d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9 not found: ID does not exist" containerID="d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9"
Sep 16 18:59:52.321418 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.321341    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9"} err="failed to get container status \"d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9\": rpc error: code = NotFound desc = could not find container \"d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9\": container with ID starting with d0f796b071170b92a8c6e2dcdfe7b3807e374f69c61d90b30daa3e726adefab9 not found: ID does not exist"
Sep 16 18:59:52.321418 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.321358    2567 scope.go:117] "RemoveContainer" containerID="b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f"
Sep 16 18:59:52.321650 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:59:52.321633    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f\": container with ID starting with b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f not found: ID does not exist" containerID="b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f"
Sep 16 18:59:52.321695 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.321657    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f"} err="failed to get container status \"b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f\": rpc error: code = NotFound desc = could not find container \"b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f\": container with ID starting with b1a314ddbd4537b2f2a96bc8f4218cedd775a1f043b77a4df8e3726c2a931d7f not found: ID does not exist"
Sep 16 18:59:52.321695 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.321672    2567 scope.go:117] "RemoveContainer" containerID="c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53"
Sep 16 18:59:52.321984 ip-10-0-143-139 kubenswrapper[2567]: E0916 18:59:52.321962    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53\": container with ID starting with c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53 not found: ID does not exist" containerID="c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53"
Sep 16 18:59:52.322091 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.321986    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53"} err="failed to get container status \"c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53\": rpc error: code = NotFound desc = could not find container \"c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53\": container with ID starting with c28a61d6a6291e302a4d0a94c06940a6e39de371984c80265b18d8f66e848f53 not found: ID does not exist"
Sep 16 18:59:52.324194 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.324174    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-n29rd"]
Sep 16 18:59:52.872828 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.872799    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"]
Sep 16 18:59:52.873167 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873152    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kube-rbac-proxy"
Sep 16 18:59:52.873221 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873170    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kube-rbac-proxy"
Sep 16 18:59:52.873221 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873185    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kube-rbac-proxy"
Sep 16 18:59:52.873221 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873191    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kube-rbac-proxy"
Sep 16 18:59:52.873221 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873199    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container"
Sep 16 18:59:52.873221 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873205    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container"
Sep 16 18:59:52.873221 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873217    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container"
Sep 16 18:59:52.873221 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873222    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container"
Sep 16 18:59:52.873434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873231    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="storage-initializer"
Sep 16 18:59:52.873434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873236    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="storage-initializer"
Sep 16 18:59:52.873434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873247    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="storage-initializer"
Sep 16 18:59:52.873434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873251    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="storage-initializer"
Sep 16 18:59:52.873434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873309    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kube-rbac-proxy"
Sep 16 18:59:52.873434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873317    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d397a5ad-2723-4616-8d4e-e404da2a4c0d" containerName="kserve-container"
Sep 16 18:59:52.873434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873324    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kube-rbac-proxy"
Sep 16 18:59:52.873434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.873329    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" containerName="kserve-container"
Sep 16 18:59:52.878235 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.878215    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:52.882042 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.882016    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 18:59:52.882042 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.882035    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:59:52.882219 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.882148    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\""
Sep 16 18:59:52.882426 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.882405    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:59:52.883410 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.883389    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 16 18:59:52.883536 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.883461    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:59:52.883536 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.883392    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:59:52.889770 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.889752    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"]
Sep 16 18:59:52.958411 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.958383    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:52.958562 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.958433    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:52.958562 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.958504    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:52.958674 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.958564    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:52.958674 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.958598    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkkm\" (UniqueName: \"kubernetes.io/projected/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-api-access-9qkkm\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:52.972597 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.972571    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"]
Sep 16 18:59:52.975977 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.975960    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:52.978753 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.978733    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 18:59:52.991465 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:52.991433    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"]
Sep 16 18:59:53.059394 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.059765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059405    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkkm\" (UniqueName: \"kubernetes.io/projected/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-api-access-9qkkm\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.059765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.059765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.059765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059527    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfbg8\" (UniqueName: \"kubernetes.io/projected/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-api-access-mfbg8\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.059765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059543    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.059765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.059765 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059725    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.060218 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059785    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.060218 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059818    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.060218 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.059971    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.060382 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.060289    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.060434 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.060410    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.062194 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.062171    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.070625 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.070601    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkkm\" (UniqueName: \"kubernetes.io/projected/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-api-access-9qkkm\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-6ggqc\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.160427 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.160348    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfbg8\" (UniqueName: \"kubernetes.io/projected/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-api-access-mfbg8\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.160427 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.160385    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.160427 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.160423    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.160651 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.160463    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.160651 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.160500    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.160809 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.160789    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.161316 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.161294    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.161354 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.161294    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.162820 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.162802    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.171800 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.171775    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfbg8\" (UniqueName: \"kubernetes.io/projected/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-api-access-mfbg8\") pod \"isvc-canary-stop-predictor-57d5c586ff-vgfv2\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.190976 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.190941    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:53.289244 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.289214    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:53.334011 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.333964    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4f732d8-64d9-4693-b946-f9d08bc9023a" path="/var/lib/kubelet/pods/e4f732d8-64d9-4693-b946-f9d08bc9023a/volumes"
Sep 16 18:59:53.471821 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.471788    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"]
Sep 16 18:59:53.474121 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:59:53.474079    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e7d1d5_0352_46ce_b23d_5a6ef4d84403.slice/crio-bbb4c7233729f33edef0584344f0ad121f3ebd7d039ca5562de3d49e711e589b WatchSource:0}: Error finding container bbb4c7233729f33edef0584344f0ad121f3ebd7d039ca5562de3d49e711e589b: Status 404 returned error can't find the container with id bbb4c7233729f33edef0584344f0ad121f3ebd7d039ca5562de3d49e711e589b
Sep 16 18:59:53.565587 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:53.565563    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"]
Sep 16 18:59:53.567222 ip-10-0-143-139 kubenswrapper[2567]: W0916 18:59:53.567195    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ae91b3_ae2d_4787_a8f4_cf94421c89d5.slice/crio-142aeda8408eb58a0e00e94923009078402d3bd018eb57024cf027c8543f3dfb WatchSource:0}: Error finding container 142aeda8408eb58a0e00e94923009078402d3bd018eb57024cf027c8543f3dfb: Status 404 returned error can't find the container with id 142aeda8408eb58a0e00e94923009078402d3bd018eb57024cf027c8543f3dfb
Sep 16 18:59:54.308392 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:54.308352    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" event={"ID":"57e7d1d5-0352-46ce-b23d-5a6ef4d84403","Type":"ContainerStarted","Data":"99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1"}
Sep 16 18:59:54.308807 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:54.308413    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" event={"ID":"57e7d1d5-0352-46ce-b23d-5a6ef4d84403","Type":"ContainerStarted","Data":"bbb4c7233729f33edef0584344f0ad121f3ebd7d039ca5562de3d49e711e589b"}
Sep 16 18:59:54.309892 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:54.309866    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" event={"ID":"41ae91b3-ae2d-4787-a8f4-cf94421c89d5","Type":"ContainerStarted","Data":"d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653"}
Sep 16 18:59:54.310012 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:54.309922    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" event={"ID":"41ae91b3-ae2d-4787-a8f4-cf94421c89d5","Type":"ContainerStarted","Data":"142aeda8408eb58a0e00e94923009078402d3bd018eb57024cf027c8543f3dfb"}
Sep 16 18:59:58.329504 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:58.329470    2567 generic.go:356] "Generic (PLEG): container finished" podID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerID="d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653" exitCode=0
Sep 16 18:59:58.329892 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:58.329544    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" event={"ID":"41ae91b3-ae2d-4787-a8f4-cf94421c89d5","Type":"ContainerDied","Data":"d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653"}
Sep 16 18:59:58.331261 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:58.331237    2567 generic.go:356] "Generic (PLEG): container finished" podID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerID="99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1" exitCode=0
Sep 16 18:59:58.331360 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:58.331281    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" event={"ID":"57e7d1d5-0352-46ce-b23d-5a6ef4d84403","Type":"ContainerDied","Data":"99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1"}
Sep 16 18:59:59.337188 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:59.337147    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" event={"ID":"57e7d1d5-0352-46ce-b23d-5a6ef4d84403","Type":"ContainerStarted","Data":"745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440"}
Sep 16 18:59:59.337591 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:59.337218    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" event={"ID":"57e7d1d5-0352-46ce-b23d-5a6ef4d84403","Type":"ContainerStarted","Data":"87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8"}
Sep 16 18:59:59.337591 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:59.337403    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 18:59:59.339305 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:59.339278    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" event={"ID":"41ae91b3-ae2d-4787-a8f4-cf94421c89d5","Type":"ContainerStarted","Data":"31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729"}
Sep 16 18:59:59.339430 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:59.339328    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" event={"ID":"41ae91b3-ae2d-4787-a8f4-cf94421c89d5","Type":"ContainerStarted","Data":"9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb"}
Sep 16 18:59:59.339514 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:59.339497    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 18:59:59.359862 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:59.359805    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podStartSLOduration=3.407988435 podStartE2EDuration="7.359790655s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.95180222s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:59.358951138 +0000 UTC m=+866.547397823" watchObservedRunningTime="2026-09-16 18:59:59.359790655 +0000 UTC m=+866.548237337"
Sep 16 18:59:59.384958 ip-10-0-143-139 kubenswrapper[2567]: I0916 18:59:59.384907    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podStartSLOduration=3.433912786 podStartE2EDuration="7.384892785s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.950979999s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:59.382703691 +0000 UTC m=+866.571150373" watchObservedRunningTime="2026-09-16 18:59:59.384892785 +0000 UTC m=+866.573339519"
Sep 16 19:00:00.345797 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:00.345759    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 19:00:00.346253 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:00.345827    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 19:00:00.346602 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:00.346575    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:00.346739 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:00.346719    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:00:01.350100 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:01.350059    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:00:01.350558 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:01.350221    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:06.356021 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:06.355989    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 19:00:06.356667 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:06.356618    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:00:06.356990 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:06.356972    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 19:00:06.357457 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:06.357436    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:13.104004 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.103964    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"]
Sep 16 19:00:13.108133 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.108098    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.110966 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.110944    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-serving-cert\""
Sep 16 19:00:13.111065 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.110988    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-9ebe2-kube-rbac-proxy-sar-config\""
Sep 16 19:00:13.119837 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.119814    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"]
Sep 16 19:00:13.233520 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.233484    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"]
Sep 16 19:00:13.237738 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.237718    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.240872 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.240854    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-serving-cert\""
Sep 16 19:00:13.241072 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.241058    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-9ebe2-kube-rbac-proxy-sar-config\""
Sep 16 19:00:13.242028 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.242006    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.242171 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.242087    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.242233 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.242172    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlxc\" (UniqueName: \"kubernetes.io/projected/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-api-access-jvlxc\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.242233 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.242198    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.242362 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.242232    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.249999 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.249980    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"]
Sep 16 19:00:13.343064 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.343064 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343057    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3519cdf2-2758-4e15-a22d-dd7488c0c456-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.343309 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343085    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.343309 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343130    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqcxz\" (UniqueName: \"kubernetes.io/projected/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-api-access-lqcxz\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.343309 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343158    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlxc\" (UniqueName: \"kubernetes.io/projected/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-api-access-jvlxc\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.343309 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.343309 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343192    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3519cdf2-2758-4e15-a22d-dd7488c0c456-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.343309 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343245    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.343309 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343283    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.343675 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343343    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.343675 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343447    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.343776 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343746    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.343865 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.343845    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.345624 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.345606    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.353530 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.353505    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlxc\" (UniqueName: \"kubernetes.io/projected/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-api-access-jvlxc\") pod \"isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.419847 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.419821    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:13.444591 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.444566    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3519cdf2-2758-4e15-a22d-dd7488c0c456-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.444727 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.444606    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.444727 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.444650    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqcxz\" (UniqueName: \"kubernetes.io/projected/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-api-access-lqcxz\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.444727 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.444674    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3519cdf2-2758-4e15-a22d-dd7488c0c456-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.444906 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:00:13.444724    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-9ebe2-predictor-serving-cert" not found
Sep 16 19:00:13.444906 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:00:13.444810    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3519cdf2-2758-4e15-a22d-dd7488c0c456-proxy-tls podName:3519cdf2-2758-4e15-a22d-dd7488c0c456 nodeName:}" failed. No retries permitted until 2026-09-16 19:00:13.944788424 +0000 UTC m=+881.133235097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3519cdf2-2758-4e15-a22d-dd7488c0c456-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" (UID: "3519cdf2-2758-4e15-a22d-dd7488c0c456") : secret "isvc-xgboost-graph-raw-hpa-9ebe2-predictor-serving-cert" not found
Sep 16 19:00:13.444906 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.444851    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.445626 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.445125    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3519cdf2-2758-4e15-a22d-dd7488c0c456-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.445626 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.445443    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.445776 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.445756    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.455271 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.455224    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqcxz\" (UniqueName: \"kubernetes.io/projected/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-api-access-lqcxz\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.701940 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.701910    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"]
Sep 16 19:00:13.703797 ip-10-0-143-139 kubenswrapper[2567]: W0916 19:00:13.703770    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ef0593_7912_4e59_a2cc_8a9f936fa0b9.slice/crio-01d40447b95078386c92a9571d617771bf4a59a0680a62a072e20f6b9a2cf972 WatchSource:0}: Error finding container 01d40447b95078386c92a9571d617771bf4a59a0680a62a072e20f6b9a2cf972: Status 404 returned error can't find the container with id 01d40447b95078386c92a9571d617771bf4a59a0680a62a072e20f6b9a2cf972
Sep 16 19:00:13.948405 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.948359    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3519cdf2-2758-4e15-a22d-dd7488c0c456-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:13.950741 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:13.950714    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3519cdf2-2758-4e15-a22d-dd7488c0c456-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:14.149132 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:14.149073    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:14.416747 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:14.416658    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" event={"ID":"13ef0593-7912-4e59-a2cc-8a9f936fa0b9","Type":"ContainerStarted","Data":"f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20"}
Sep 16 19:00:14.416888 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:14.416774    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" event={"ID":"13ef0593-7912-4e59-a2cc-8a9f936fa0b9","Type":"ContainerStarted","Data":"01d40447b95078386c92a9571d617771bf4a59a0680a62a072e20f6b9a2cf972"}
Sep 16 19:00:14.427477 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:14.427450    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"]
Sep 16 19:00:14.427692 ip-10-0-143-139 kubenswrapper[2567]: W0916 19:00:14.427668    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3519cdf2_2758_4e15_a22d_dd7488c0c456.slice/crio-62a7e74b262996a59eb171dd654ab5c53d33bccd7de084e871aa934e145b6bb7 WatchSource:0}: Error finding container 62a7e74b262996a59eb171dd654ab5c53d33bccd7de084e871aa934e145b6bb7: Status 404 returned error can't find the container with id 62a7e74b262996a59eb171dd654ab5c53d33bccd7de084e871aa934e145b6bb7
Sep 16 19:00:15.422463 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:15.422426    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" event={"ID":"3519cdf2-2758-4e15-a22d-dd7488c0c456","Type":"ContainerStarted","Data":"349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362"}
Sep 16 19:00:15.422953 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:15.422495    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" event={"ID":"3519cdf2-2758-4e15-a22d-dd7488c0c456","Type":"ContainerStarted","Data":"62a7e74b262996a59eb171dd654ab5c53d33bccd7de084e871aa934e145b6bb7"}
Sep 16 19:00:16.357532 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:16.357497    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:00:16.357532 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:16.357509    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:18.439607 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:18.439577    2567 generic.go:356] "Generic (PLEG): container finished" podID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerID="349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362" exitCode=0
Sep 16 19:00:18.439941 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:18.439651    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" event={"ID":"3519cdf2-2758-4e15-a22d-dd7488c0c456","Type":"ContainerDied","Data":"349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362"}
Sep 16 19:00:18.441149 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:18.441130    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:00:18.441445 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:18.441421    2567 generic.go:356] "Generic (PLEG): container finished" podID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerID="f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20" exitCode=0
Sep 16 19:00:18.441556 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:18.441457    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" event={"ID":"13ef0593-7912-4e59-a2cc-8a9f936fa0b9","Type":"ContainerDied","Data":"f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20"}
Sep 16 19:00:19.454655 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:19.454619    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" event={"ID":"13ef0593-7912-4e59-a2cc-8a9f936fa0b9","Type":"ContainerStarted","Data":"6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570"}
Sep 16 19:00:19.455066 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:19.454699    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" event={"ID":"13ef0593-7912-4e59-a2cc-8a9f936fa0b9","Type":"ContainerStarted","Data":"6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f"}
Sep 16 19:00:19.455794 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:19.455769    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:19.455925 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:19.455827    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:19.457783 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:19.457512    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:00:19.482248 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:19.482191    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podStartSLOduration=2.789838062 podStartE2EDuration="6.482172017s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.692333955s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:19.479921545 +0000 UTC m=+886.668368240" watchObservedRunningTime="2026-09-16 19:00:19.482172017 +0000 UTC m=+886.670618701"
Sep 16 19:00:20.460841 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:20.460790    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:00:25.466427 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:25.465986    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:00:25.472301 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:25.467399    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:00:26.356869 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:26.356784    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:00:26.357654 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:26.357550    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:35.467928 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:35.467889    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:00:36.356964 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:36.356921    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:00:36.357389 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:36.357355    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:36.568359 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:36.568330    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 19:00:36.568759 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:36.568338    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 19:00:37.557967 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:37.557932    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" event={"ID":"3519cdf2-2758-4e15-a22d-dd7488c0c456","Type":"ContainerStarted","Data":"249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444"}
Sep 16 19:00:37.558157 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:37.558003    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" event={"ID":"3519cdf2-2758-4e15-a22d-dd7488c0c456","Type":"ContainerStarted","Data":"a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c"}
Sep 16 19:00:37.558417 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:37.558355    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:37.558566 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:37.558545    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:37.559516 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:37.559491    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:00:37.592192 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:37.592148    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podStartSLOduration=2.504630213 podStartE2EDuration="24.59213478s" totalImagesPullingTime="18.214468847s" totalInitContainerRuntime="3.87303572s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:37.588486644 +0000 UTC m=+904.776933324" watchObservedRunningTime="2026-09-16 19:00:37.59213478 +0000 UTC m=+904.780581455"
Sep 16 19:00:38.563721 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:38.563679    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:00:43.568371 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:43.568343    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:00:43.568816 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:43.568791    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:00:45.467977 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:45.467944    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:00:46.356605 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:46.356560    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:00:46.357833 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:46.357795    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:53.568873 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:53.568830    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:00:55.468231 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:55.468191    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:00:56.357242 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:56.357195    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:00:56.357448 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:00:56.357398    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:01:03.569382 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:03.569345    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:01:05.467859 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:05.467820    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:01:06.356847 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:06.356805    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:01:06.357368 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:06.357336    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:01:13.569659 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:13.569623    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:01:15.467695 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:15.467655    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:01:16.357385 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:16.357332    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:01:16.357385 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:16.357346    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:01:17.329026 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:17.328987    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:01:23.569750 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:23.569716    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:01:25.468364 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:25.468323    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:01:26.358190 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:26.358095    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 19:01:26.489808 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:26.489768    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"]
Sep 16 19:01:26.490299 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:26.490123    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container" containerID="cri-o://9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb" gracePeriod=30
Sep 16 19:01:26.490299 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:26.490143    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kube-rbac-proxy" containerID="cri-o://31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729" gracePeriod=30
Sep 16 19:01:26.490940 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:26.490905    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 19:01:26.802767 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:26.802731    2567 generic.go:356] "Generic (PLEG): container finished" podID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerID="31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729" exitCode=2
Sep 16 19:01:26.802974 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:26.802801    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" event={"ID":"41ae91b3-ae2d-4787-a8f4-cf94421c89d5","Type":"ContainerDied","Data":"31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729"}
Sep 16 19:01:31.329833 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.329809    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 19:01:31.448657 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.448590    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-proxy-tls\") pod \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") "
Sep 16 19:01:31.448657 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.448642    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-cabundle-cert\") pod \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") "
Sep 16 19:01:31.448862 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.448667    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfbg8\" (UniqueName: \"kubernetes.io/projected/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-api-access-mfbg8\") pod \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") "
Sep 16 19:01:31.448862 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.448693    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kserve-provision-location\") pod \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") "
Sep 16 19:01:31.448862 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.448749    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-rbac-proxy-sar-config\") pod \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\" (UID: \"41ae91b3-ae2d-4787-a8f4-cf94421c89d5\") "
Sep 16 19:01:31.449028 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.448943    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41ae91b3-ae2d-4787-a8f4-cf94421c89d5" (UID: "41ae91b3-ae2d-4787-a8f4-cf94421c89d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:31.449080 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.449025    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41ae91b3-ae2d-4787-a8f4-cf94421c89d5" (UID: "41ae91b3-ae2d-4787-a8f4-cf94421c89d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:31.449182 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.449164    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:01:31.449228 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.449179    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41ae91b3-ae2d-4787-a8f4-cf94421c89d5" (UID: "41ae91b3-ae2d-4787-a8f4-cf94421c89d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:31.449228 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.449189    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:01:31.450710 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.450685    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41ae91b3-ae2d-4787-a8f4-cf94421c89d5" (UID: "41ae91b3-ae2d-4787-a8f4-cf94421c89d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:31.450836 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.450734    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-api-access-mfbg8" (OuterVolumeSpecName: "kube-api-access-mfbg8") pod "41ae91b3-ae2d-4787-a8f4-cf94421c89d5" (UID: "41ae91b3-ae2d-4787-a8f4-cf94421c89d5"). InnerVolumeSpecName "kube-api-access-mfbg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:31.550572 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.550539    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfbg8\" (UniqueName: \"kubernetes.io/projected/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-api-access-mfbg8\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:01:31.550572 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.550575    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:01:31.550790 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.550590    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ae91b3-ae2d-4787-a8f4-cf94421c89d5-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:01:31.827434 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.827351    2567 generic.go:356] "Generic (PLEG): container finished" podID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerID="9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb" exitCode=0
Sep 16 19:01:31.827434 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.827410    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"
Sep 16 19:01:31.827623 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.827404    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" event={"ID":"41ae91b3-ae2d-4787-a8f4-cf94421c89d5","Type":"ContainerDied","Data":"9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb"}
Sep 16 19:01:31.827623 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.827549    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2" event={"ID":"41ae91b3-ae2d-4787-a8f4-cf94421c89d5","Type":"ContainerDied","Data":"142aeda8408eb58a0e00e94923009078402d3bd018eb57024cf027c8543f3dfb"}
Sep 16 19:01:31.827623 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.827578    2567 scope.go:117] "RemoveContainer" containerID="31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729"
Sep 16 19:01:31.836840 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.836823    2567 scope.go:117] "RemoveContainer" containerID="9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb"
Sep 16 19:01:31.845970 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.845954    2567 scope.go:117] "RemoveContainer" containerID="d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653"
Sep 16 19:01:31.853979 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.853957    2567 scope.go:117] "RemoveContainer" containerID="31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729"
Sep 16 19:01:31.854277 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:01:31.854259    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729\": container with ID starting with 31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729 not found: ID does not exist" containerID="31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729"
Sep 16 19:01:31.854358 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.854290    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729"} err="failed to get container status \"31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729\": rpc error: code = NotFound desc = could not find container \"31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729\": container with ID starting with 31a6c585ac9d816a0db04b97e2872517585eb3302ad31bd55dfa1789a8d6d729 not found: ID does not exist"
Sep 16 19:01:31.854358 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.854315    2567 scope.go:117] "RemoveContainer" containerID="9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb"
Sep 16 19:01:31.854583 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:01:31.854565    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb\": container with ID starting with 9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb not found: ID does not exist" containerID="9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb"
Sep 16 19:01:31.854643 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.854590    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb"} err="failed to get container status \"9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb\": rpc error: code = NotFound desc = could not find container \"9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb\": container with ID starting with 9ee66e27cb537ffb16635e53fc134ef37152da3339bc0e7f35787d5ff04441eb not found: ID does not exist"
Sep 16 19:01:31.854643 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.854618    2567 scope.go:117] "RemoveContainer" containerID="d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653"
Sep 16 19:01:31.854956 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:01:31.854937    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653\": container with ID starting with d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653 not found: ID does not exist" containerID="d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653"
Sep 16 19:01:31.854997 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.854960    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653"} err="failed to get container status \"d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653\": rpc error: code = NotFound desc = could not find container \"d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653\": container with ID starting with d7a7c9fafffa3fabb4e2077140bd844635a86905677f4e4456361a76dd842653 not found: ID does not exist"
Sep 16 19:01:31.864105 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.864067    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"]
Sep 16 19:01:31.869126 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:31.869086    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vgfv2"]
Sep 16 19:01:32.885969 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:32.885929    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"]
Sep 16 19:01:32.886431 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:32.886242    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" containerID="cri-o://87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8" gracePeriod=30
Sep 16 19:01:32.886431 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:32.886296    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kube-rbac-proxy" containerID="cri-o://745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440" gracePeriod=30
Sep 16 19:01:33.335040 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:33.334972    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" path="/var/lib/kubelet/pods/41ae91b3-ae2d-4787-a8f4-cf94421c89d5/volumes"
Sep 16 19:01:33.568937 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:33.568904    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:01:33.839309 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:33.839278    2567 generic.go:356] "Generic (PLEG): container finished" podID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerID="745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440" exitCode=2
Sep 16 19:01:33.839309 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:33.839313    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" event={"ID":"57e7d1d5-0352-46ce-b23d-5a6ef4d84403","Type":"ContainerDied","Data":"745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440"}
Sep 16 19:01:35.467782 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:35.467744    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:01:36.350754 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:36.350707    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:01:36.358353 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:36.358312    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:01:37.721234 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.721212    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 19:01:37.807914 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.807828    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-cabundle-cert\") pod \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") "
Sep 16 19:01:37.807914 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.807879    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qkkm\" (UniqueName: \"kubernetes.io/projected/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-api-access-9qkkm\") pod \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") "
Sep 16 19:01:37.807914 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.807914    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kserve-provision-location\") pod \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") "
Sep 16 19:01:37.808156 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.807945    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-proxy-tls\") pod \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") "
Sep 16 19:01:37.808156 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.808011    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-rbac-proxy-sar-config\") pod \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\" (UID: \"57e7d1d5-0352-46ce-b23d-5a6ef4d84403\") "
Sep 16 19:01:37.808273 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.808226    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57e7d1d5-0352-46ce-b23d-5a6ef4d84403" (UID: "57e7d1d5-0352-46ce-b23d-5a6ef4d84403"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:37.808377 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.808291    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57e7d1d5-0352-46ce-b23d-5a6ef4d84403" (UID: "57e7d1d5-0352-46ce-b23d-5a6ef4d84403"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:37.808377 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.808365    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:01:37.808520 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.808464    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57e7d1d5-0352-46ce-b23d-5a6ef4d84403" (UID: "57e7d1d5-0352-46ce-b23d-5a6ef4d84403"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:37.809956 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.809935    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-api-access-9qkkm" (OuterVolumeSpecName: "kube-api-access-9qkkm") pod "57e7d1d5-0352-46ce-b23d-5a6ef4d84403" (UID: "57e7d1d5-0352-46ce-b23d-5a6ef4d84403"). InnerVolumeSpecName "kube-api-access-9qkkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:37.809956 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.809944    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57e7d1d5-0352-46ce-b23d-5a6ef4d84403" (UID: "57e7d1d5-0352-46ce-b23d-5a6ef4d84403"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:37.861563 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.861535    2567 generic.go:356] "Generic (PLEG): container finished" podID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerID="87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8" exitCode=0
Sep 16 19:01:37.861673 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.861587    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"
Sep 16 19:01:37.861673 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.861614    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" event={"ID":"57e7d1d5-0352-46ce-b23d-5a6ef4d84403","Type":"ContainerDied","Data":"87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8"}
Sep 16 19:01:37.861673 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.861670    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc" event={"ID":"57e7d1d5-0352-46ce-b23d-5a6ef4d84403","Type":"ContainerDied","Data":"bbb4c7233729f33edef0584344f0ad121f3ebd7d039ca5562de3d49e711e589b"}
Sep 16 19:01:37.861831 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.861697    2567 scope.go:117] "RemoveContainer" containerID="745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440"
Sep 16 19:01:37.870171 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.870150    2567 scope.go:117] "RemoveContainer" containerID="87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8"
Sep 16 19:01:37.877284 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.877266    2567 scope.go:117] "RemoveContainer" containerID="99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1"
Sep 16 19:01:37.884478 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.884462    2567 scope.go:117] "RemoveContainer" containerID="745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440"
Sep 16 19:01:37.884746 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:01:37.884728    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440\": container with ID starting with 745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440 not found: ID does not exist" containerID="745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440"
Sep 16 19:01:37.884787 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.884754    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440"} err="failed to get container status \"745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440\": rpc error: code = NotFound desc = could not find container \"745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440\": container with ID starting with 745ade4ea8f6b12b08aaf545fb6346c90e8dea401611192e080ae2ee07eff440 not found: ID does not exist"
Sep 16 19:01:37.884787 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.884770    2567 scope.go:117] "RemoveContainer" containerID="87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8"
Sep 16 19:01:37.885005 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:01:37.884992    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8\": container with ID starting with 87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8 not found: ID does not exist" containerID="87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8"
Sep 16 19:01:37.885042 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.885009    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8"} err="failed to get container status \"87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8\": rpc error: code = NotFound desc = could not find container \"87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8\": container with ID starting with 87e124e32a71652492dc9245d7d33f45d62f661437c82b451b6d1a30e559a2e8 not found: ID does not exist"
Sep 16 19:01:37.885042 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.885020    2567 scope.go:117] "RemoveContainer" containerID="99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1"
Sep 16 19:01:37.885224 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:01:37.885209    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1\": container with ID starting with 99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1 not found: ID does not exist" containerID="99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1"
Sep 16 19:01:37.885270 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.885229    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1"} err="failed to get container status \"99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1\": rpc error: code = NotFound desc = could not find container \"99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1\": container with ID starting with 99dc3415b873aa23a180c5808ca42944442709d3ec0e8b6c65dd2bef483b70a1 not found: ID does not exist"
Sep 16 19:01:37.888980 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.888958    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"]
Sep 16 19:01:37.893176 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.893156    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-6ggqc"]
Sep 16 19:01:37.908955 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.908939    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:01:37.909030 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.908957    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:01:37.909030 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.908966    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkkm\" (UniqueName: \"kubernetes.io/projected/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kube-api-access-9qkkm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:01:37.909030 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:37.908977    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e7d1d5-0352-46ce-b23d-5a6ef4d84403-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:01:39.339711 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:39.339670    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" path="/var/lib/kubelet/pods/57e7d1d5-0352-46ce-b23d-5a6ef4d84403/volumes"
Sep 16 19:01:43.570146 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:43.570103    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:01:45.467899 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:01:45.467866    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:02:13.369657 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:13.369582    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"]
Sep 16 19:02:13.370058 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:13.369878    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" containerID="cri-o://6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f" gracePeriod=30
Sep 16 19:02:13.370058 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:13.369898    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kube-rbac-proxy" containerID="cri-o://6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570" gracePeriod=30
Sep 16 19:02:13.451664 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:13.451628    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"]
Sep 16 19:02:13.451980 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:13.451918    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container" containerID="cri-o://a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c" gracePeriod=30
Sep 16 19:02:13.452139 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:13.451934    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kube-rbac-proxy" containerID="cri-o://249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444" gracePeriod=30
Sep 16 19:02:13.563875 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:13.563844    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 16 19:02:13.569331 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:13.569304    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:02:14.033412 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:14.033380    2567 generic.go:356] "Generic (PLEG): container finished" podID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerID="249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444" exitCode=2
Sep 16 19:02:14.033587 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:14.033459    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" event={"ID":"3519cdf2-2758-4e15-a22d-dd7488c0c456","Type":"ContainerDied","Data":"249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444"}
Sep 16 19:02:14.035631 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:14.035612    2567 generic.go:356] "Generic (PLEG): container finished" podID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerID="6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570" exitCode=2
Sep 16 19:02:14.035709 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:14.035646    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" event={"ID":"13ef0593-7912-4e59-a2cc-8a9f936fa0b9","Type":"ContainerDied","Data":"6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570"}
Sep 16 19:02:15.461828 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:15.461789    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 19:02:15.468344 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:15.468313    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:02:16.999064 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:16.999042    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:02:17.039949 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.039881    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3519cdf2-2758-4e15-a22d-dd7488c0c456-proxy-tls\") pod \"3519cdf2-2758-4e15-a22d-dd7488c0c456\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") "
Sep 16 19:02:17.040080 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.039952    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-cabundle-cert\") pod \"3519cdf2-2758-4e15-a22d-dd7488c0c456\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") "
Sep 16 19:02:17.040080 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.039991    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-rbac-proxy-sar-config\") pod \"3519cdf2-2758-4e15-a22d-dd7488c0c456\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") "
Sep 16 19:02:17.040080 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.040056    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqcxz\" (UniqueName: \"kubernetes.io/projected/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-api-access-lqcxz\") pod \"3519cdf2-2758-4e15-a22d-dd7488c0c456\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") "
Sep 16 19:02:17.040283 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.040103    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3519cdf2-2758-4e15-a22d-dd7488c0c456-kserve-provision-location\") pod \"3519cdf2-2758-4e15-a22d-dd7488c0c456\" (UID: \"3519cdf2-2758-4e15-a22d-dd7488c0c456\") "
Sep 16 19:02:17.040450 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.040425    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3519cdf2-2758-4e15-a22d-dd7488c0c456" (UID: "3519cdf2-2758-4e15-a22d-dd7488c0c456"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:17.040516 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.040433    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3519cdf2-2758-4e15-a22d-dd7488c0c456" (UID: "3519cdf2-2758-4e15-a22d-dd7488c0c456"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:17.040516 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.040434    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3519cdf2-2758-4e15-a22d-dd7488c0c456-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3519cdf2-2758-4e15-a22d-dd7488c0c456" (UID: "3519cdf2-2758-4e15-a22d-dd7488c0c456"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:17.041942 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.041920    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3519cdf2-2758-4e15-a22d-dd7488c0c456-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3519cdf2-2758-4e15-a22d-dd7488c0c456" (UID: "3519cdf2-2758-4e15-a22d-dd7488c0c456"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:17.042039 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.042023    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-api-access-lqcxz" (OuterVolumeSpecName: "kube-api-access-lqcxz") pod "3519cdf2-2758-4e15-a22d-dd7488c0c456" (UID: "3519cdf2-2758-4e15-a22d-dd7488c0c456"). InnerVolumeSpecName "kube-api-access-lqcxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:17.052643 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.052560    2567 generic.go:356] "Generic (PLEG): container finished" podID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerID="a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c" exitCode=0
Sep 16 19:02:17.052643 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.052632    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"
Sep 16 19:02:17.052826 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.052728    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" event={"ID":"3519cdf2-2758-4e15-a22d-dd7488c0c456","Type":"ContainerDied","Data":"a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c"}
Sep 16 19:02:17.052826 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.052783    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt" event={"ID":"3519cdf2-2758-4e15-a22d-dd7488c0c456","Type":"ContainerDied","Data":"62a7e74b262996a59eb171dd654ab5c53d33bccd7de084e871aa934e145b6bb7"}
Sep 16 19:02:17.052826 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.052819    2567 scope.go:117] "RemoveContainer" containerID="249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444"
Sep 16 19:02:17.063661 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.063635    2567 scope.go:117] "RemoveContainer" containerID="a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c"
Sep 16 19:02:17.072347 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.072325    2567 scope.go:117] "RemoveContainer" containerID="349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362"
Sep 16 19:02:17.081146 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.081101    2567 scope.go:117] "RemoveContainer" containerID="249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444"
Sep 16 19:02:17.081470 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:02:17.081438    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444\": container with ID starting with 249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444 not found: ID does not exist" containerID="249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444"
Sep 16 19:02:17.081626 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.081483    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444"} err="failed to get container status \"249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444\": rpc error: code = NotFound desc = could not find container \"249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444\": container with ID starting with 249e4174d5fa2ddd74379fc1a4e314c9a001c7e8dd03eda046599f891b31a444 not found: ID does not exist"
Sep 16 19:02:17.081626 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.081508    2567 scope.go:117] "RemoveContainer" containerID="a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c"
Sep 16 19:02:17.081894 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:02:17.081872    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c\": container with ID starting with a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c not found: ID does not exist" containerID="a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c"
Sep 16 19:02:17.081994 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.081903    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c"} err="failed to get container status \"a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c\": rpc error: code = NotFound desc = could not find container \"a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c\": container with ID starting with a981ed67b12fa8276a0e4a8d2d0ef5a92ce0b381b2b0a98a08be17bc3b5b768c not found: ID does not exist"
Sep 16 19:02:17.081994 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.081923    2567 scope.go:117] "RemoveContainer" containerID="349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362"
Sep 16 19:02:17.082311 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:02:17.082238    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362\": container with ID starting with 349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362 not found: ID does not exist" containerID="349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362"
Sep 16 19:02:17.082311 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.082266    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362"} err="failed to get container status \"349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362\": rpc error: code = NotFound desc = could not find container \"349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362\": container with ID starting with 349c81b34f45bccac037667b9a1a8e0b66a8508f46d7a8c70318469cf08a3362 not found: ID does not exist"
Sep 16 19:02:17.084261 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.084241    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"]
Sep 16 19:02:17.085863 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.085844    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-9ebe2-predictor-c767cd969-8fmxt"]
Sep 16 19:02:17.141711 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.141686    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqcxz\" (UniqueName: \"kubernetes.io/projected/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-api-access-lqcxz\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:02:17.141711 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.141708    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3519cdf2-2758-4e15-a22d-dd7488c0c456-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:02:17.141835 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.141719    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3519cdf2-2758-4e15-a22d-dd7488c0c456-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:02:17.141835 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.141727    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:02:17.141835 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.141735    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3519cdf2-2758-4e15-a22d-dd7488c0c456-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:02:17.334319 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:17.334241    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" path="/var/lib/kubelet/pods/3519cdf2-2758-4e15-a22d-dd7488c0c456/volumes"
Sep 16 19:02:18.011013 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.010988    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:02:18.050956 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.050918    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kserve-provision-location\") pod \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") "
Sep 16 19:02:18.051147 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.050964    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-rbac-proxy-sar-config\") pod \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") "
Sep 16 19:02:18.051147 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.051007    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-cabundle-cert\") pod \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") "
Sep 16 19:02:18.051147 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.051072    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-proxy-tls\") pod \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") "
Sep 16 19:02:18.051147 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.051138    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvlxc\" (UniqueName: \"kubernetes.io/projected/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-api-access-jvlxc\") pod \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\" (UID: \"13ef0593-7912-4e59-a2cc-8a9f936fa0b9\") "
Sep 16 19:02:18.051394 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.051275    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13ef0593-7912-4e59-a2cc-8a9f936fa0b9" (UID: "13ef0593-7912-4e59-a2cc-8a9f936fa0b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:18.051394 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.051384    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:02:18.051515 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.051418    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13ef0593-7912-4e59-a2cc-8a9f936fa0b9" (UID: "13ef0593-7912-4e59-a2cc-8a9f936fa0b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:18.051515 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.051425    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13ef0593-7912-4e59-a2cc-8a9f936fa0b9" (UID: "13ef0593-7912-4e59-a2cc-8a9f936fa0b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:18.053183 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.053156    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-api-access-jvlxc" (OuterVolumeSpecName: "kube-api-access-jvlxc") pod "13ef0593-7912-4e59-a2cc-8a9f936fa0b9" (UID: "13ef0593-7912-4e59-a2cc-8a9f936fa0b9"). InnerVolumeSpecName "kube-api-access-jvlxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:18.053183 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.053174    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13ef0593-7912-4e59-a2cc-8a9f936fa0b9" (UID: "13ef0593-7912-4e59-a2cc-8a9f936fa0b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:18.061147 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.061105    2567 generic.go:356] "Generic (PLEG): container finished" podID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerID="6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f" exitCode=0
Sep 16 19:02:18.061238 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.061158    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" event={"ID":"13ef0593-7912-4e59-a2cc-8a9f936fa0b9","Type":"ContainerDied","Data":"6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f"}
Sep 16 19:02:18.061238 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.061190    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"
Sep 16 19:02:18.061311 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.061240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht" event={"ID":"13ef0593-7912-4e59-a2cc-8a9f936fa0b9","Type":"ContainerDied","Data":"01d40447b95078386c92a9571d617771bf4a59a0680a62a072e20f6b9a2cf972"}
Sep 16 19:02:18.061311 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.061268    2567 scope.go:117] "RemoveContainer" containerID="6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570"
Sep 16 19:02:18.070458 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.070419    2567 scope.go:117] "RemoveContainer" containerID="6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f"
Sep 16 19:02:18.077856 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.077835    2567 scope.go:117] "RemoveContainer" containerID="f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20"
Sep 16 19:02:18.084977 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.084961    2567 scope.go:117] "RemoveContainer" containerID="6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570"
Sep 16 19:02:18.085357 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:02:18.085334    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570\": container with ID starting with 6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570 not found: ID does not exist" containerID="6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570"
Sep 16 19:02:18.085442 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.085360    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570"} err="failed to get container status \"6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570\": rpc error: code = NotFound desc = could not find container \"6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570\": container with ID starting with 6499f578c7df5e99711d6408ba4daf2c3e470dbdff375b69885e0f24b3ee7570 not found: ID does not exist"
Sep 16 19:02:18.085442 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.085382    2567 scope.go:117] "RemoveContainer" containerID="6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f"
Sep 16 19:02:18.085636 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:02:18.085620    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f\": container with ID starting with 6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f not found: ID does not exist" containerID="6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f"
Sep 16 19:02:18.085677 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.085642    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f"} err="failed to get container status \"6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f\": rpc error: code = NotFound desc = could not find container \"6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f\": container with ID starting with 6f96fed2e4dce3cd9bc8b552c7db60b017046107deebce2a56ccc66d2e305b1f not found: ID does not exist"
Sep 16 19:02:18.085677 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.085663    2567 scope.go:117] "RemoveContainer" containerID="f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20"
Sep 16 19:02:18.085903 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:02:18.085887    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20\": container with ID starting with f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20 not found: ID does not exist" containerID="f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20"
Sep 16 19:02:18.085956 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.085906    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20"} err="failed to get container status \"f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20\": rpc error: code = NotFound desc = could not find container \"f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20\": container with ID starting with f7c1f2fa4a4d83842211df0f8e865a4d7ecf946dcd70f8c950fd7ef207141b20 not found: ID does not exist"
Sep 16 19:02:18.090700 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.090680    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"]
Sep 16 19:02:18.094945 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.094926    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9ebe2-predictor-79d95b6d9d-j7fht"]
Sep 16 19:02:18.152608 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.152583    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:02:18.152608 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.152608    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvlxc\" (UniqueName: \"kubernetes.io/projected/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-api-access-jvlxc\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:02:18.152774 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.152618    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:02:18.152774 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:18.152628    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13ef0593-7912-4e59-a2cc-8a9f936fa0b9-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:02:19.333608 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:02:19.333577    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" path="/var/lib/kubelet/pods/13ef0593-7912-4e59-a2cc-8a9f936fa0b9/volumes"
Sep 16 19:03:28.303027 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.302989    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"]
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303339    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303350    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303360    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="storage-initializer"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303366    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="storage-initializer"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303381    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303387    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303396    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303401    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303407    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="storage-initializer"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303414    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="storage-initializer"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303421    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303425    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303432 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303437    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303443    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303451    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303456    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303470    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="storage-initializer"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303475    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="storage-initializer"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303480    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303485    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303492    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303497    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303512    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="storage-initializer"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303518    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="storage-initializer"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303571    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kserve-container"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303577    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3519cdf2-2758-4e15-a22d-dd7488c0c456" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303584    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kserve-container"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303589    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ae91b3-ae2d-4787-a8f4-cf94421c89d5" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303595    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kube-rbac-proxy"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303600    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ef0593-7912-4e59-a2cc-8a9f936fa0b9" containerName="kserve-container"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303606    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kserve-container"
Sep 16 19:03:28.303824 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.303611    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e7d1d5-0352-46ce-b23d-5a6ef4d84403" containerName="kube-rbac-proxy"
Sep 16 19:03:28.306276 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.306261    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.308963 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.308939    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:03:28.308963 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.308949    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9d9a89-predictor-serving-cert\""
Sep 16 19:03:28.310356 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.310337    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:03:28.310469 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.310347    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-9d9a89\""
Sep 16 19:03:28.310469 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.310397    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9d9a89-kube-rbac-proxy-sar-config\""
Sep 16 19:03:28.310469 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.310346    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-9d9a89-dockercfg-bffl6\""
Sep 16 19:03:28.310715 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.310696    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:28.319671 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.319653    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"]
Sep 16 19:03:28.457547 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.457516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f40af73-f97b-46dd-b9c7-31a740451009-kserve-provision-location\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.457707 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.457560    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f40af73-f97b-46dd-b9c7-31a740451009-proxy-tls\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.457707 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.457640    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-cabundle-cert\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.457707 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.457674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.457707 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.457690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccgzj\" (UniqueName: \"kubernetes.io/projected/2f40af73-f97b-46dd-b9c7-31a740451009-kube-api-access-ccgzj\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.558829 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.558735    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f40af73-f97b-46dd-b9c7-31a740451009-kserve-provision-location\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.558829 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.558778    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f40af73-f97b-46dd-b9c7-31a740451009-proxy-tls\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.559080 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.558918    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-cabundle-cert\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.559080 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.558965    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.559080 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.558990    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccgzj\" (UniqueName: \"kubernetes.io/projected/2f40af73-f97b-46dd-b9c7-31a740451009-kube-api-access-ccgzj\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.559283 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.559233    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f40af73-f97b-46dd-b9c7-31a740451009-kserve-provision-location\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.559629 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.559595    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.559715 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.559601    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-cabundle-cert\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.561255 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.561236    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f40af73-f97b-46dd-b9c7-31a740451009-proxy-tls\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.568624 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.568605    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccgzj\" (UniqueName: \"kubernetes.io/projected/2f40af73-f97b-46dd-b9c7-31a740451009-kube-api-access-ccgzj\") pod \"isvc-secondary-9d9a89-predictor-74798678d-rvg2t\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") " pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.619496 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.619471    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:28.894384 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:28.894361    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"]
Sep 16 19:03:28.896310 ip-10-0-143-139 kubenswrapper[2567]: W0916 19:03:28.896281    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f40af73_f97b_46dd_b9c7_31a740451009.slice/crio-867ce119558a48c0840fb2d15fec03a0c44c95f55083ae60b8dd119f7cb31e57 WatchSource:0}: Error finding container 867ce119558a48c0840fb2d15fec03a0c44c95f55083ae60b8dd119f7cb31e57: Status 404 returned error can't find the container with id 867ce119558a48c0840fb2d15fec03a0c44c95f55083ae60b8dd119f7cb31e57
Sep 16 19:03:29.360954 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:29.360867    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t" event={"ID":"2f40af73-f97b-46dd-b9c7-31a740451009","Type":"ContainerStarted","Data":"1b397560129f84387528356350b0d68e62f84cae2806c2820bc7061f01292918"}
Sep 16 19:03:29.360954 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:29.360930    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t" event={"ID":"2f40af73-f97b-46dd-b9c7-31a740451009","Type":"ContainerStarted","Data":"867ce119558a48c0840fb2d15fec03a0c44c95f55083ae60b8dd119f7cb31e57"}
Sep 16 19:03:35.390549 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:35.390523    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9d9a89-predictor-74798678d-rvg2t_2f40af73-f97b-46dd-b9c7-31a740451009/storage-initializer/0.log"
Sep 16 19:03:35.390962 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:35.390559    2567 generic.go:356] "Generic (PLEG): container finished" podID="2f40af73-f97b-46dd-b9c7-31a740451009" containerID="1b397560129f84387528356350b0d68e62f84cae2806c2820bc7061f01292918" exitCode=1
Sep 16 19:03:35.390962 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:35.390583    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t" event={"ID":"2f40af73-f97b-46dd-b9c7-31a740451009","Type":"ContainerDied","Data":"1b397560129f84387528356350b0d68e62f84cae2806c2820bc7061f01292918"}
Sep 16 19:03:36.396707 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:36.396680    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9d9a89-predictor-74798678d-rvg2t_2f40af73-f97b-46dd-b9c7-31a740451009/storage-initializer/0.log"
Sep 16 19:03:36.397092 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:36.396722    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t" event={"ID":"2f40af73-f97b-46dd-b9c7-31a740451009","Type":"ContainerStarted","Data":"1d0050b8171189be42465270a3f29af4b82d23a27d67b7738ba97a29cdf4bc13"}
Sep 16 19:03:41.420696 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:41.420668    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9d9a89-predictor-74798678d-rvg2t_2f40af73-f97b-46dd-b9c7-31a740451009/storage-initializer/1.log"
Sep 16 19:03:41.421102 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:41.421087    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9d9a89-predictor-74798678d-rvg2t_2f40af73-f97b-46dd-b9c7-31a740451009/storage-initializer/0.log"
Sep 16 19:03:41.421182 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:41.421139    2567 generic.go:356] "Generic (PLEG): container finished" podID="2f40af73-f97b-46dd-b9c7-31a740451009" containerID="1d0050b8171189be42465270a3f29af4b82d23a27d67b7738ba97a29cdf4bc13" exitCode=1
Sep 16 19:03:41.421221 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:41.421187    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t" event={"ID":"2f40af73-f97b-46dd-b9c7-31a740451009","Type":"ContainerDied","Data":"1d0050b8171189be42465270a3f29af4b82d23a27d67b7738ba97a29cdf4bc13"}
Sep 16 19:03:41.421270 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:41.421242    2567 scope.go:117] "RemoveContainer" containerID="1b397560129f84387528356350b0d68e62f84cae2806c2820bc7061f01292918"
Sep 16 19:03:41.421583 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:41.421552    2567 scope.go:117] "RemoveContainer" containerID="1b397560129f84387528356350b0d68e62f84cae2806c2820bc7061f01292918"
Sep 16 19:03:41.432649 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:03:41.432622    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-9d9a89-predictor-74798678d-rvg2t_kserve-ci-e2e-test(2f40af73-f97b-46dd-b9c7-31a740451009)\"" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t" podUID="2f40af73-f97b-46dd-b9c7-31a740451009"
Sep 16 19:03:41.432897 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:03:41.432869    2567 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9d9a89-predictor-74798678d-rvg2t_kserve-ci-e2e-test_2f40af73-f97b-46dd-b9c7-31a740451009_0 in pod sandbox 867ce119558a48c0840fb2d15fec03a0c44c95f55083ae60b8dd119f7cb31e57 from index: no such id: '1b397560129f84387528356350b0d68e62f84cae2806c2820bc7061f01292918'" containerID="1b397560129f84387528356350b0d68e62f84cae2806c2820bc7061f01292918"
Sep 16 19:03:41.432947 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:41.432909    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b397560129f84387528356350b0d68e62f84cae2806c2820bc7061f01292918"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9d9a89-predictor-74798678d-rvg2t_kserve-ci-e2e-test_2f40af73-f97b-46dd-b9c7-31a740451009_0 in pod sandbox 867ce119558a48c0840fb2d15fec03a0c44c95f55083ae60b8dd119f7cb31e57 from index: no such id: '1b397560129f84387528356350b0d68e62f84cae2806c2820bc7061f01292918'"
Sep 16 19:03:42.426945 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:42.426918    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9d9a89-predictor-74798678d-rvg2t_2f40af73-f97b-46dd-b9c7-31a740451009/storage-initializer/1.log"
Sep 16 19:03:46.530688 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.530655    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"]
Sep 16 19:03:46.667378 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.667352    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9d9a89-predictor-74798678d-rvg2t_2f40af73-f97b-46dd-b9c7-31a740451009/storage-initializer/1.log"
Sep 16 19:03:46.670783 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.670765    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:46.710224 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.710190    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccgzj\" (UniqueName: \"kubernetes.io/projected/2f40af73-f97b-46dd-b9c7-31a740451009-kube-api-access-ccgzj\") pod \"2f40af73-f97b-46dd-b9c7-31a740451009\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") "
Sep 16 19:03:46.710401 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.710300    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-cabundle-cert\") pod \"2f40af73-f97b-46dd-b9c7-31a740451009\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") "
Sep 16 19:03:46.710401 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.710326    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-kube-rbac-proxy-sar-config\") pod \"2f40af73-f97b-46dd-b9c7-31a740451009\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") "
Sep 16 19:03:46.710401 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.710354    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f40af73-f97b-46dd-b9c7-31a740451009-kserve-provision-location\") pod \"2f40af73-f97b-46dd-b9c7-31a740451009\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") "
Sep 16 19:03:46.710565 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.710411    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f40af73-f97b-46dd-b9c7-31a740451009-proxy-tls\") pod \"2f40af73-f97b-46dd-b9c7-31a740451009\" (UID: \"2f40af73-f97b-46dd-b9c7-31a740451009\") "
Sep 16 19:03:46.710678 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.710645    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f40af73-f97b-46dd-b9c7-31a740451009-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f40af73-f97b-46dd-b9c7-31a740451009" (UID: "2f40af73-f97b-46dd-b9c7-31a740451009"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:46.710727 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.710692    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f40af73-f97b-46dd-b9c7-31a740451009" (UID: "2f40af73-f97b-46dd-b9c7-31a740451009"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:46.710727 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.710699    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f40af73-f97b-46dd-b9c7-31a740451009" (UID: "2f40af73-f97b-46dd-b9c7-31a740451009"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:46.712289 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.712269    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f40af73-f97b-46dd-b9c7-31a740451009-kube-api-access-ccgzj" (OuterVolumeSpecName: "kube-api-access-ccgzj") pod "2f40af73-f97b-46dd-b9c7-31a740451009" (UID: "2f40af73-f97b-46dd-b9c7-31a740451009"). InnerVolumeSpecName "kube-api-access-ccgzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:46.712428 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.712407    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f40af73-f97b-46dd-b9c7-31a740451009-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f40af73-f97b-46dd-b9c7-31a740451009" (UID: "2f40af73-f97b-46dd-b9c7-31a740451009"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:46.811404 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.811328    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:03:46.811404 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.811355    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f40af73-f97b-46dd-b9c7-31a740451009-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:03:46.811404 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.811365    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f40af73-f97b-46dd-b9c7-31a740451009-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:03:46.811404 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.811375    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f40af73-f97b-46dd-b9c7-31a740451009-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:03:46.811404 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:46.811384    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccgzj\" (UniqueName: \"kubernetes.io/projected/2f40af73-f97b-46dd-b9c7-31a740451009-kube-api-access-ccgzj\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:03:47.451740 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:47.451709    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9d9a89-predictor-74798678d-rvg2t_2f40af73-f97b-46dd-b9c7-31a740451009/storage-initializer/1.log"
Sep 16 19:03:47.451907 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:47.451807    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"
Sep 16 19:03:47.451907 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:47.451810    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t" event={"ID":"2f40af73-f97b-46dd-b9c7-31a740451009","Type":"ContainerDied","Data":"867ce119558a48c0840fb2d15fec03a0c44c95f55083ae60b8dd119f7cb31e57"}
Sep 16 19:03:47.451985 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:47.451929    2567 scope.go:117] "RemoveContainer" containerID="1d0050b8171189be42465270a3f29af4b82d23a27d67b7738ba97a29cdf4bc13"
Sep 16 19:03:47.497309 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:47.497272    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"]
Sep 16 19:03:47.502448 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:47.502419    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9d9a89-predictor-74798678d-rvg2t"]
Sep 16 19:03:49.333678 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:03:49.333638    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f40af73-f97b-46dd-b9c7-31a740451009" path="/var/lib/kubelet/pods/2f40af73-f97b-46dd-b9c7-31a740451009/volumes"
Sep 16 19:04:01.939465 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.939431    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"]
Sep 16 19:04:01.939955 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.939941    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f40af73-f97b-46dd-b9c7-31a740451009" containerName="storage-initializer"
Sep 16 19:04:01.940002 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.939959    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f40af73-f97b-46dd-b9c7-31a740451009" containerName="storage-initializer"
Sep 16 19:04:01.940002 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.939987    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f40af73-f97b-46dd-b9c7-31a740451009" containerName="storage-initializer"
Sep 16 19:04:01.940002 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.939997    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f40af73-f97b-46dd-b9c7-31a740451009" containerName="storage-initializer"
Sep 16 19:04:01.940101 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.940074    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f40af73-f97b-46dd-b9c7-31a740451009" containerName="storage-initializer"
Sep 16 19:04:01.940276 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.940264    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f40af73-f97b-46dd-b9c7-31a740451009" containerName="storage-initializer"
Sep 16 19:04:01.943624 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.943606    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:01.946987 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.946959    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:04:01.947101 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.947000    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-15dec-kube-rbac-proxy-sar-config\""
Sep 16 19:04:01.947185 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.947170    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\""
Sep 16 19:04:01.947393 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.947373    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-15dec-predictor-serving-cert\""
Sep 16 19:04:01.948327 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.948310    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:04:01.948407 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.948343    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:04:01.948407 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.948348    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:04:01.960408 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:01.960379    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"]
Sep 16 19:04:02.041577 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.041541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.041741 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.041648    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9sb\" (UniqueName: \"kubernetes.io/projected/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-api-access-fz9sb\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.041741 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.041713    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-proxy-tls\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.041741 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.041737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kserve-provision-location\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.041883 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.041785    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-cabundle-cert\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.143002 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.142963    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9sb\" (UniqueName: \"kubernetes.io/projected/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-api-access-fz9sb\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.143161 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.143023    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-proxy-tls\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.143216 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.143163    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kserve-provision-location\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.143216 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.143200    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-cabundle-cert\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.143288 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.143236    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.143597 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.143574    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kserve-provision-location\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.143846 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.143828    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-cabundle-cert\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.143895 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.143880    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.145459 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.145442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-proxy-tls\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.152984 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.152959    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9sb\" (UniqueName: \"kubernetes.io/projected/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-api-access-fz9sb\") pod \"raw-sklearn-15dec-predictor-65988d47cd-bnd4v\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") " pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.254055 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.253975    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:02.533838 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:02.533813    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"]
Sep 16 19:04:02.535798 ip-10-0-143-139 kubenswrapper[2567]: W0916 19:04:02.535771    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b3b5ad_9fa1_4615_97e2_583d2c5260a3.slice/crio-4fa1942413741a378aebdcadb8f356b3cd418cfacaa889a1bfcd96679d25d10c WatchSource:0}: Error finding container 4fa1942413741a378aebdcadb8f356b3cd418cfacaa889a1bfcd96679d25d10c: Status 404 returned error can't find the container with id 4fa1942413741a378aebdcadb8f356b3cd418cfacaa889a1bfcd96679d25d10c
Sep 16 19:04:03.523707 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:03.523669    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" event={"ID":"89b3b5ad-9fa1-4615-97e2-583d2c5260a3","Type":"ContainerStarted","Data":"4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49"}
Sep 16 19:04:03.524089 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:03.523729    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" event={"ID":"89b3b5ad-9fa1-4615-97e2-583d2c5260a3","Type":"ContainerStarted","Data":"4fa1942413741a378aebdcadb8f356b3cd418cfacaa889a1bfcd96679d25d10c"}
Sep 16 19:04:06.538339 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:06.538306    2567 generic.go:356] "Generic (PLEG): container finished" podID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerID="4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49" exitCode=0
Sep 16 19:04:06.538733 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:06.538380    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" event={"ID":"89b3b5ad-9fa1-4615-97e2-583d2c5260a3","Type":"ContainerDied","Data":"4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49"}
Sep 16 19:04:07.544814 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:07.544780    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" event={"ID":"89b3b5ad-9fa1-4615-97e2-583d2c5260a3","Type":"ContainerStarted","Data":"63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3"}
Sep 16 19:04:07.545216 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:07.544843    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" event={"ID":"89b3b5ad-9fa1-4615-97e2-583d2c5260a3","Type":"ContainerStarted","Data":"ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b"}
Sep 16 19:04:07.545216 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:07.545029    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:07.571437 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:07.571398    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podStartSLOduration=2.7301657219999997 podStartE2EDuration="6.571385553s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841219831s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:07.570001869 +0000 UTC m=+1114.758448551" watchObservedRunningTime="2026-09-16 19:04:07.571385553 +0000 UTC m=+1114.759832236"
Sep 16 19:04:08.549928 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:08.549897    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:08.551326 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:08.551298    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:04:09.554288 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:09.554248    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:04:14.559292 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:14.559260    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:04:14.559850 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:14.559815    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:04:24.560128 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:24.560068    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:04:34.559718 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:34.559676    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:04:44.560746 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:44.560701    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:04:54.559847 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:04:54.559782    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:05:04.560678 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:04.560598    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:05:14.560471 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:14.560428    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:05:23.334585 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:23.334558    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:05:32.006611 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:32.006579    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"]
Sep 16 19:05:32.007064 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:32.006838    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" containerID="cri-o://ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b" gracePeriod=30
Sep 16 19:05:32.007064 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:32.006882    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kube-rbac-proxy" containerID="cri-o://63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3" gracePeriod=30
Sep 16 19:05:32.932340 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:32.932307    2567 generic.go:356] "Generic (PLEG): container finished" podID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerID="63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3" exitCode=2
Sep 16 19:05:32.932511 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:32.932376    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" event={"ID":"89b3b5ad-9fa1-4615-97e2-583d2c5260a3","Type":"ContainerDied","Data":"63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3"}
Sep 16 19:05:33.332083 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:33.331990    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:05:34.555239 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:34.555188    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 16 19:05:36.607146 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.607102    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 19:05:36.608415 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.608394    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 19:05:36.648312 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.648291    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:05:36.762702 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.762634    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-cabundle-cert\") pod \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") "
Sep 16 19:05:36.762702 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.762693    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kserve-provision-location\") pod \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") "
Sep 16 19:05:36.762921 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.762778    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz9sb\" (UniqueName: \"kubernetes.io/projected/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-api-access-fz9sb\") pod \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") "
Sep 16 19:05:36.762921 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.762834    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-rbac-proxy-sar-config\") pod \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") "
Sep 16 19:05:36.762921 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.762863    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-proxy-tls\") pod \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\" (UID: \"89b3b5ad-9fa1-4615-97e2-583d2c5260a3\") "
Sep 16 19:05:36.763338 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.763035    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89b3b5ad-9fa1-4615-97e2-583d2c5260a3" (UID: "89b3b5ad-9fa1-4615-97e2-583d2c5260a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:36.763338 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.763157    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89b3b5ad-9fa1-4615-97e2-583d2c5260a3" (UID: "89b3b5ad-9fa1-4615-97e2-583d2c5260a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:36.763338 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.763230    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89b3b5ad-9fa1-4615-97e2-583d2c5260a3" (UID: "89b3b5ad-9fa1-4615-97e2-583d2c5260a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:36.764865 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.764831    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-api-access-fz9sb" (OuterVolumeSpecName: "kube-api-access-fz9sb") pod "89b3b5ad-9fa1-4615-97e2-583d2c5260a3" (UID: "89b3b5ad-9fa1-4615-97e2-583d2c5260a3"). InnerVolumeSpecName "kube-api-access-fz9sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:36.764944 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.764897    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89b3b5ad-9fa1-4615-97e2-583d2c5260a3" (UID: "89b3b5ad-9fa1-4615-97e2-583d2c5260a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:36.864533 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.864498    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz9sb\" (UniqueName: \"kubernetes.io/projected/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-api-access-fz9sb\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:05:36.864533 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.864529    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:05:36.864711 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.864543    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:05:36.864711 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.864557    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:05:36.864711 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.864568    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b3b5ad-9fa1-4615-97e2-583d2c5260a3-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:05:36.951734 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.951698    2567 generic.go:356] "Generic (PLEG): container finished" podID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerID="ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b" exitCode=0
Sep 16 19:05:36.951866 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.951758    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"
Sep 16 19:05:36.951866 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.951753    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" event={"ID":"89b3b5ad-9fa1-4615-97e2-583d2c5260a3","Type":"ContainerDied","Data":"ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b"}
Sep 16 19:05:36.951942 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.951889    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v" event={"ID":"89b3b5ad-9fa1-4615-97e2-583d2c5260a3","Type":"ContainerDied","Data":"4fa1942413741a378aebdcadb8f356b3cd418cfacaa889a1bfcd96679d25d10c"}
Sep 16 19:05:36.951942 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.951928    2567 scope.go:117] "RemoveContainer" containerID="63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3"
Sep 16 19:05:36.961238 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.961221    2567 scope.go:117] "RemoveContainer" containerID="ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b"
Sep 16 19:05:36.970170 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.970146    2567 scope.go:117] "RemoveContainer" containerID="4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49"
Sep 16 19:05:36.976243 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.976222    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"]
Sep 16 19:05:36.979072 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.979054    2567 scope.go:117] "RemoveContainer" containerID="63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3"
Sep 16 19:05:36.979400 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:05:36.979379    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3\": container with ID starting with 63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3 not found: ID does not exist" containerID="63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3"
Sep 16 19:05:36.979484 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.979407    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3"} err="failed to get container status \"63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3\": rpc error: code = NotFound desc = could not find container \"63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3\": container with ID starting with 63d46af3e26dc147a8deb917384456a892e5072b16bcedccdb79b7d1540a07a3 not found: ID does not exist"
Sep 16 19:05:36.979484 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.979424    2567 scope.go:117] "RemoveContainer" containerID="ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b"
Sep 16 19:05:36.979667 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:05:36.979651    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b\": container with ID starting with ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b not found: ID does not exist" containerID="ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b"
Sep 16 19:05:36.979710 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.979670    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b"} err="failed to get container status \"ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b\": rpc error: code = NotFound desc = could not find container \"ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b\": container with ID starting with ff239ddd5011d6cad9977de1979e308404758599b771808a54d7b6171880867b not found: ID does not exist"
Sep 16 19:05:36.979710 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.979683    2567 scope.go:117] "RemoveContainer" containerID="4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49"
Sep 16 19:05:36.979925 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:05:36.979904    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49\": container with ID starting with 4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49 not found: ID does not exist" containerID="4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49"
Sep 16 19:05:36.980010 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.979932    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49"} err="failed to get container status \"4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49\": rpc error: code = NotFound desc = could not find container \"4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49\": container with ID starting with 4540a742943f6751bd923663c0c5248dbc730c75b86f133f97c780fbe8457e49 not found: ID does not exist"
Sep 16 19:05:36.980798 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:36.980769    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-15dec-predictor-65988d47cd-bnd4v"]
Sep 16 19:05:37.334350 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:05:37.334311    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" path="/var/lib/kubelet/pods/89b3b5ad-9fa1-4615-97e2-583d2c5260a3/volumes"
Sep 16 19:07:04.132352 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.132319    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"]
Sep 16 19:07:04.132798 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.132729    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container"
Sep 16 19:07:04.132798 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.132742    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container"
Sep 16 19:07:04.132798 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.132755    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="storage-initializer"
Sep 16 19:07:04.132798 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.132761    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="storage-initializer"
Sep 16 19:07:04.132798 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.132780    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kube-rbac-proxy"
Sep 16 19:07:04.132798 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.132785    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kube-rbac-proxy"
Sep 16 19:07:04.132990 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.132839    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kube-rbac-proxy"
Sep 16 19:07:04.132990 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.132848    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b3b5ad-9fa1-4615-97e2-583d2c5260a3" containerName="kserve-container"
Sep 16 19:07:04.135881 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.135864    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.139464 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.139439    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:07:04.139603 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.139499    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:07:04.139603 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.139560    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:07:04.139830 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.139814    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:07:04.140360 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.140346    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 19:07:04.140886 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.140868    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\""
Sep 16 19:07:04.140945 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.140898    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:07:04.156634 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.156615    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"]
Sep 16 19:07:04.190355 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.190333    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf40490a-a08e-4345-8cb4-65640725cfb2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.190473 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.190374    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.190473 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.190461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf40490a-a08e-4345-8cb4-65640725cfb2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.190562 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.190508    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv9jb\" (UniqueName: \"kubernetes.io/projected/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-api-access-kv9jb\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.190562 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.190537    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.291856 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.291796    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9jb\" (UniqueName: \"kubernetes.io/projected/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-api-access-kv9jb\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.291856 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.291842    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.292081 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.291891    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf40490a-a08e-4345-8cb4-65640725cfb2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.292081 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.291937    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.292081 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.292011    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf40490a-a08e-4345-8cb4-65640725cfb2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.292433 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.292413    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf40490a-a08e-4345-8cb4-65640725cfb2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.292553 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.292535    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.292598 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.292580    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.294359 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.294341    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf40490a-a08e-4345-8cb4-65640725cfb2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.301983 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.301962    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9jb\" (UniqueName: \"kubernetes.io/projected/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-api-access-kv9jb\") pod \"isvc-canary-create-v2-predictor-847844f798-cg9rs\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.446461 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.446379    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:04.725081 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:04.725048    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"]
Sep 16 19:07:04.727374 ip-10-0-143-139 kubenswrapper[2567]: W0916 19:07:04.727349    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf40490a_a08e_4345_8cb4_65640725cfb2.slice/crio-70f7efc4783afaae4a322638860e23241c771455844e9446448e4905c2d16d71 WatchSource:0}: Error finding container 70f7efc4783afaae4a322638860e23241c771455844e9446448e4905c2d16d71: Status 404 returned error can't find the container with id 70f7efc4783afaae4a322638860e23241c771455844e9446448e4905c2d16d71
Sep 16 19:07:05.349282 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:05.349252    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" event={"ID":"bf40490a-a08e-4345-8cb4-65640725cfb2","Type":"ContainerStarted","Data":"572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed"}
Sep 16 19:07:05.349640 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:05.349309    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" event={"ID":"bf40490a-a08e-4345-8cb4-65640725cfb2","Type":"ContainerStarted","Data":"70f7efc4783afaae4a322638860e23241c771455844e9446448e4905c2d16d71"}
Sep 16 19:07:09.368482 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:09.368448    2567 generic.go:356] "Generic (PLEG): container finished" podID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerID="572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed" exitCode=0
Sep 16 19:07:09.368923 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:09.368487    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" event={"ID":"bf40490a-a08e-4345-8cb4-65640725cfb2","Type":"ContainerDied","Data":"572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed"}
Sep 16 19:07:10.375216 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:10.375180    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" event={"ID":"bf40490a-a08e-4345-8cb4-65640725cfb2","Type":"ContainerStarted","Data":"39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2"}
Sep 16 19:07:10.375676 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:10.375249    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" event={"ID":"bf40490a-a08e-4345-8cb4-65640725cfb2","Type":"ContainerStarted","Data":"a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf"}
Sep 16 19:07:10.375676 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:10.375495    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:10.399838 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:10.399790    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podStartSLOduration=2.727163182 podStartE2EDuration="6.399773421s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.672610239s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:10.396661619 +0000 UTC m=+1297.585108302" watchObservedRunningTime="2026-09-16 19:07:10.399773421 +0000 UTC m=+1297.588220156"
Sep 16 19:07:11.379581 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:11.379543    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:11.380808 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:11.380780    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:07:12.383819 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:12.383777    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:07:17.388209 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:17.388179    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:07:17.388847 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:17.388817    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:07:27.389244 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:27.389211    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:07:37.388730 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:37.388689    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:07:47.389045 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:47.389000    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:07:57.388846 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:07:57.388761    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:08:07.389700 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:07.389663    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:08:17.389621 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:17.389586    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:08:23.705088 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.705051    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f4dc4d8bd-hhzf7"]
Sep 16 19:08:23.709149 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.709123    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.721880 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.721853    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f4dc4d8bd-hhzf7"]
Sep 16 19:08:23.770798 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.770772    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-service-ca\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.770973 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.770829    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-trusted-ca-bundle\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.770973 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.770869    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-oauth-serving-cert\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.770973 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.770903    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqzr\" (UniqueName: \"kubernetes.io/projected/423fe583-eac8-41b6-b7a1-b5a37d421752-kube-api-access-rtqzr\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.770973 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.770945    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-console-config\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.771193 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.770995    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/423fe583-eac8-41b6-b7a1-b5a37d421752-console-serving-cert\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.771193 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.771030    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/423fe583-eac8-41b6-b7a1-b5a37d421752-console-oauth-config\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.871868 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.871835    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-trusted-ca-bundle\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.871868 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.871873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-oauth-serving-cert\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.872082 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.871902    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqzr\" (UniqueName: \"kubernetes.io/projected/423fe583-eac8-41b6-b7a1-b5a37d421752-kube-api-access-rtqzr\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.872082 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.871926    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-console-config\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.872082 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.871959    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/423fe583-eac8-41b6-b7a1-b5a37d421752-console-serving-cert\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.872082 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.872004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/423fe583-eac8-41b6-b7a1-b5a37d421752-console-oauth-config\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.872321 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.872099    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-service-ca\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.872689 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.872661    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-oauth-serving-cert\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.872837 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.872771    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-console-config\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.872837 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.872826    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-trusted-ca-bundle\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.872938 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.872826    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/423fe583-eac8-41b6-b7a1-b5a37d421752-service-ca\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.874324 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.874300    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/423fe583-eac8-41b6-b7a1-b5a37d421752-console-serving-cert\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.874437 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.874419    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/423fe583-eac8-41b6-b7a1-b5a37d421752-console-oauth-config\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:23.888133 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:23.888098    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqzr\" (UniqueName: \"kubernetes.io/projected/423fe583-eac8-41b6-b7a1-b5a37d421752-kube-api-access-rtqzr\") pod \"console-6f4dc4d8bd-hhzf7\" (UID: \"423fe583-eac8-41b6-b7a1-b5a37d421752\") " pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:24.019173 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:24.019073    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:24.299740 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:24.299691    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f4dc4d8bd-hhzf7"]
Sep 16 19:08:24.302377 ip-10-0-143-139 kubenswrapper[2567]: W0916 19:08:24.302345    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423fe583_eac8_41b6_b7a1_b5a37d421752.slice/crio-d0a396bab81d4e7835065dc1768c6b49b183b45912824ebb6cf0c061e95088cf WatchSource:0}: Error finding container d0a396bab81d4e7835065dc1768c6b49b183b45912824ebb6cf0c061e95088cf: Status 404 returned error can't find the container with id d0a396bab81d4e7835065dc1768c6b49b183b45912824ebb6cf0c061e95088cf
Sep 16 19:08:24.715726 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:24.715684    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4dc4d8bd-hhzf7" event={"ID":"423fe583-eac8-41b6-b7a1-b5a37d421752","Type":"ContainerStarted","Data":"87d5066ee729e8a64b288df84465f866b902b671bdf01c462c56a2bf22eaa38f"}
Sep 16 19:08:24.716140 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:24.715770    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4dc4d8bd-hhzf7" event={"ID":"423fe583-eac8-41b6-b7a1-b5a37d421752","Type":"ContainerStarted","Data":"d0a396bab81d4e7835065dc1768c6b49b183b45912824ebb6cf0c061e95088cf"}
Sep 16 19:08:24.740469 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:24.740425    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f4dc4d8bd-hhzf7" podStartSLOduration=1.740411249 podStartE2EDuration="1.740411249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:24.738028994 +0000 UTC m=+1371.926475694" watchObservedRunningTime="2026-09-16 19:08:24.740411249 +0000 UTC m=+1371.928857930"
Sep 16 19:08:27.389271 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:27.389238    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:08:34.019319 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:34.019227    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:34.019319 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:34.019293    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:34.023844 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:34.023822    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:34.189525 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:34.189495    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"]
Sep 16 19:08:34.189767 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:34.189723    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" containerID="cri-o://a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf" gracePeriod=30
Sep 16 19:08:34.189871 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:34.189771    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kube-rbac-proxy" containerID="cri-o://39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2" gracePeriod=30
Sep 16 19:08:34.768171 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:34.768136    2567 generic.go:356] "Generic (PLEG): container finished" podID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerID="39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2" exitCode=2
Sep 16 19:08:34.768361 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:34.768207    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" event={"ID":"bf40490a-a08e-4345-8cb4-65640725cfb2","Type":"ContainerDied","Data":"39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2"}
Sep 16 19:08:34.772157 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:34.772135    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f4dc4d8bd-hhzf7"
Sep 16 19:08:34.825957 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:34.825927    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b4b6db4d5-cvqm8"]
Sep 16 19:08:37.384159 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:37.384098    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 19:08:37.389455 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:37.389430    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:08:38.728713 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.728692    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:08:38.786567 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.786535    2567 generic.go:356] "Generic (PLEG): container finished" podID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerID="a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf" exitCode=0
Sep 16 19:08:38.786567 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.786572    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" event={"ID":"bf40490a-a08e-4345-8cb4-65640725cfb2","Type":"ContainerDied","Data":"a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf"}
Sep 16 19:08:38.786802 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.786599    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"
Sep 16 19:08:38.786802 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.786620    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs" event={"ID":"bf40490a-a08e-4345-8cb4-65640725cfb2","Type":"ContainerDied","Data":"70f7efc4783afaae4a322638860e23241c771455844e9446448e4905c2d16d71"}
Sep 16 19:08:38.786802 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.786648    2567 scope.go:117] "RemoveContainer" containerID="39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2"
Sep 16 19:08:38.794471 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.794461    2567 scope.go:117] "RemoveContainer" containerID="a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf"
Sep 16 19:08:38.801520 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.801506    2567 scope.go:117] "RemoveContainer" containerID="572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed"
Sep 16 19:08:38.808386 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.808371    2567 scope.go:117] "RemoveContainer" containerID="39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2"
Sep 16 19:08:38.808618 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:08:38.808601    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2\": container with ID starting with 39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2 not found: ID does not exist" containerID="39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2"
Sep 16 19:08:38.808673 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.808625    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2"} err="failed to get container status \"39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2\": rpc error: code = NotFound desc = could not find container \"39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2\": container with ID starting with 39403f9e7ece78fb7246c16afd9adccd6def5ac9aabb2546dd6cd041758644b2 not found: ID does not exist"
Sep 16 19:08:38.808673 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.808640    2567 scope.go:117] "RemoveContainer" containerID="a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf"
Sep 16 19:08:38.808837 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:08:38.808823    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf\": container with ID starting with a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf not found: ID does not exist" containerID="a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf"
Sep 16 19:08:38.808886 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.808841    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf"} err="failed to get container status \"a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf\": rpc error: code = NotFound desc = could not find container \"a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf\": container with ID starting with a9d0ddd4f563604207cdc08949b84536f87c10f896fc38f5627245ffd00bdfdf not found: ID does not exist"
Sep 16 19:08:38.808886 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.808853    2567 scope.go:117] "RemoveContainer" containerID="572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed"
Sep 16 19:08:38.809044 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:08:38.809030    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed\": container with ID starting with 572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed not found: ID does not exist" containerID="572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed"
Sep 16 19:08:38.809083 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.809046    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed"} err="failed to get container status \"572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed\": rpc error: code = NotFound desc = could not find container \"572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed\": container with ID starting with 572264518f3efad27c77f72e05af55425cb5ec1fe565c5a150007bba0a2ca8ed not found: ID does not exist"
Sep 16 19:08:38.811387 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.811372    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf40490a-a08e-4345-8cb4-65640725cfb2-kserve-provision-location\") pod \"bf40490a-a08e-4345-8cb4-65640725cfb2\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") "
Sep 16 19:08:38.811442 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.811396    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-rbac-proxy-sar-config\") pod \"bf40490a-a08e-4345-8cb4-65640725cfb2\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") "
Sep 16 19:08:38.811442 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.811417    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf40490a-a08e-4345-8cb4-65640725cfb2-proxy-tls\") pod \"bf40490a-a08e-4345-8cb4-65640725cfb2\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") "
Sep 16 19:08:38.811520 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.811447    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv9jb\" (UniqueName: \"kubernetes.io/projected/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-api-access-kv9jb\") pod \"bf40490a-a08e-4345-8cb4-65640725cfb2\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") "
Sep 16 19:08:38.811520 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.811494    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-cabundle-cert\") pod \"bf40490a-a08e-4345-8cb4-65640725cfb2\" (UID: \"bf40490a-a08e-4345-8cb4-65640725cfb2\") "
Sep 16 19:08:38.811740 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.811719    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf40490a-a08e-4345-8cb4-65640725cfb2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf40490a-a08e-4345-8cb4-65640725cfb2" (UID: "bf40490a-a08e-4345-8cb4-65640725cfb2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:38.811854 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.811748    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bf40490a-a08e-4345-8cb4-65640725cfb2" (UID: "bf40490a-a08e-4345-8cb4-65640725cfb2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:38.811922 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.811901    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bf40490a-a08e-4345-8cb4-65640725cfb2" (UID: "bf40490a-a08e-4345-8cb4-65640725cfb2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:38.813372 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.813353    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf40490a-a08e-4345-8cb4-65640725cfb2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf40490a-a08e-4345-8cb4-65640725cfb2" (UID: "bf40490a-a08e-4345-8cb4-65640725cfb2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:38.813457 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.813404    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-api-access-kv9jb" (OuterVolumeSpecName: "kube-api-access-kv9jb") pod "bf40490a-a08e-4345-8cb4-65640725cfb2" (UID: "bf40490a-a08e-4345-8cb4-65640725cfb2"). InnerVolumeSpecName "kube-api-access-kv9jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:38.912482 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.912444    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf40490a-a08e-4345-8cb4-65640725cfb2-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:08:38.912482 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.912482    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:08:38.912482 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.912493    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf40490a-a08e-4345-8cb4-65640725cfb2-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:08:38.912680 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.912503    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv9jb\" (UniqueName: \"kubernetes.io/projected/bf40490a-a08e-4345-8cb4-65640725cfb2-kube-api-access-kv9jb\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:08:38.912680 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:38.912512    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf40490a-a08e-4345-8cb4-65640725cfb2-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:08:39.110582 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:39.110540    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"]
Sep 16 19:08:39.114009 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:39.113986    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-cg9rs"]
Sep 16 19:08:39.333868 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:39.333832    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" path="/var/lib/kubelet/pods/bf40490a-a08e-4345-8cb4-65640725cfb2/volumes"
Sep 16 19:08:59.850507 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:08:59.850442    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b4b6db4d5-cvqm8" podUID="acf20a86-bdc2-45f3-9e01-86cfca2c5485" containerName="console" containerID="cri-o://629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11" gracePeriod=15
Sep 16 19:09:00.095594 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.095565    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 19:09:00.196483 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.196452    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-serving-cert\") pod \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") "
Sep 16 19:09:00.196644 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.196526    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xw62\" (UniqueName: \"kubernetes.io/projected/acf20a86-bdc2-45f3-9e01-86cfca2c5485-kube-api-access-7xw62\") pod \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") "
Sep 16 19:09:00.196644 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.196549    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-service-ca\") pod \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") "
Sep 16 19:09:00.196644 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.196580    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-oauth-serving-cert\") pod \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") "
Sep 16 19:09:00.196644 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.196604    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-oauth-config\") pod \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") "
Sep 16 19:09:00.196644 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.196625    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-config\") pod \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") "
Sep 16 19:09:00.196889 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.196675    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-trusted-ca-bundle\") pod \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\" (UID: \"acf20a86-bdc2-45f3-9e01-86cfca2c5485\") "
Sep 16 19:09:00.197061 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.197014    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "acf20a86-bdc2-45f3-9e01-86cfca2c5485" (UID: "acf20a86-bdc2-45f3-9e01-86cfca2c5485"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:00.197219 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.197178    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "acf20a86-bdc2-45f3-9e01-86cfca2c5485" (UID: "acf20a86-bdc2-45f3-9e01-86cfca2c5485"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:00.197219 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.197070    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-config" (OuterVolumeSpecName: "console-config") pod "acf20a86-bdc2-45f3-9e01-86cfca2c5485" (UID: "acf20a86-bdc2-45f3-9e01-86cfca2c5485"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:00.197324 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.197254    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-service-ca" (OuterVolumeSpecName: "service-ca") pod "acf20a86-bdc2-45f3-9e01-86cfca2c5485" (UID: "acf20a86-bdc2-45f3-9e01-86cfca2c5485"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:00.198724 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.198692    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "acf20a86-bdc2-45f3-9e01-86cfca2c5485" (UID: "acf20a86-bdc2-45f3-9e01-86cfca2c5485"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:00.199195 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.199168    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "acf20a86-bdc2-45f3-9e01-86cfca2c5485" (UID: "acf20a86-bdc2-45f3-9e01-86cfca2c5485"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:00.199280 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.199214    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf20a86-bdc2-45f3-9e01-86cfca2c5485-kube-api-access-7xw62" (OuterVolumeSpecName: "kube-api-access-7xw62") pod "acf20a86-bdc2-45f3-9e01-86cfca2c5485" (UID: "acf20a86-bdc2-45f3-9e01-86cfca2c5485"). InnerVolumeSpecName "kube-api-access-7xw62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:00.298090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.298056    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:09:00.298090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.298085    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-trusted-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:09:00.298090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.298095    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:09:00.298342 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.298106    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xw62\" (UniqueName: \"kubernetes.io/projected/acf20a86-bdc2-45f3-9e01-86cfca2c5485-kube-api-access-7xw62\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:09:00.298342 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.298143    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:09:00.298342 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.298151    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acf20a86-bdc2-45f3-9e01-86cfca2c5485-oauth-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:09:00.298342 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.298159    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acf20a86-bdc2-45f3-9e01-86cfca2c5485-console-oauth-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:09:00.895563 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.895530    2567 generic.go:356] "Generic (PLEG): container finished" podID="acf20a86-bdc2-45f3-9e01-86cfca2c5485" containerID="629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11" exitCode=0
Sep 16 19:09:00.895971 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.895574    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4b6db4d5-cvqm8"
Sep 16 19:09:00.895971 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.895610    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4b6db4d5-cvqm8" event={"ID":"acf20a86-bdc2-45f3-9e01-86cfca2c5485","Type":"ContainerDied","Data":"629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11"}
Sep 16 19:09:00.895971 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.895651    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4b6db4d5-cvqm8" event={"ID":"acf20a86-bdc2-45f3-9e01-86cfca2c5485","Type":"ContainerDied","Data":"74d814616d00414945f5f03a434b17687892c6265d900e0871123a4ac8e38a8a"}
Sep 16 19:09:00.895971 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.895670    2567 scope.go:117] "RemoveContainer" containerID="629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11"
Sep 16 19:09:00.905044 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.905023    2567 scope.go:117] "RemoveContainer" containerID="629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11"
Sep 16 19:09:00.905390 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:09:00.905371    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11\": container with ID starting with 629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11 not found: ID does not exist" containerID="629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11"
Sep 16 19:09:00.905449 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.905399    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11"} err="failed to get container status \"629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11\": rpc error: code = NotFound desc = could not find container \"629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11\": container with ID starting with 629a167c93852011fa7e4efcee07119fd68f1353defc17350b5ab5e102babd11 not found: ID does not exist"
Sep 16 19:09:00.919157 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.919131    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b4b6db4d5-cvqm8"]
Sep 16 19:09:00.924003 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:00.923979    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b4b6db4d5-cvqm8"]
Sep 16 19:09:01.335120 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:09:01.335037    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acf20a86-bdc2-45f3-9e01-86cfca2c5485" path="/var/lib/kubelet/pods/acf20a86-bdc2-45f3-9e01-86cfca2c5485/volumes"
Sep 16 19:10:09.513650 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.513612    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"]
Sep 16 19:10:09.514090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.513963    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kube-rbac-proxy"
Sep 16 19:10:09.514090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.513973    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kube-rbac-proxy"
Sep 16 19:10:09.514090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.513988    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container"
Sep 16 19:10:09.514090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.513994    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container"
Sep 16 19:10:09.514090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.514003    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf20a86-bdc2-45f3-9e01-86cfca2c5485" containerName="console"
Sep 16 19:10:09.514090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.514009    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf20a86-bdc2-45f3-9e01-86cfca2c5485" containerName="console"
Sep 16 19:10:09.514090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.514016    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="storage-initializer"
Sep 16 19:10:09.514090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.514021    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="storage-initializer"
Sep 16 19:10:09.514090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.514079    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kserve-container"
Sep 16 19:10:09.514090 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.514089    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf20a86-bdc2-45f3-9e01-86cfca2c5485" containerName="console"
Sep 16 19:10:09.514419 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.514097    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf40490a-a08e-4345-8cb4-65640725cfb2" containerName="kube-rbac-proxy"
Sep 16 19:10:09.517222 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.517205    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.521272 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.521249    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:10:09.521402 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.521308    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:10:09.521666 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.521649    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:10:09.522763 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.522746    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:10:09.523395 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.523380    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:10:09.524362 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.524340    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:10:09.524447 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.524343    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\""
Sep 16 19:10:09.553025 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.553001    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"]
Sep 16 19:10:09.709445 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.709414    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.709612 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.709476    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4173ef65-0d8d-4867-840a-0750921ecfeb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.709612 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.709508    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwz7\" (UniqueName: \"kubernetes.io/projected/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-api-access-dpwz7\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.709612 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.709531    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.709612 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.709564    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4173ef65-0d8d-4867-840a-0750921ecfeb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.810050 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.809966    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4173ef65-0d8d-4867-840a-0750921ecfeb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.810050 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.810003    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwz7\" (UniqueName: \"kubernetes.io/projected/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-api-access-dpwz7\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.810050 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.810030    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.810304 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.810062    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4173ef65-0d8d-4867-840a-0750921ecfeb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.810304 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.810098    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.810523 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.810502    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4173ef65-0d8d-4867-840a-0750921ecfeb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.810709 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.810692    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.810851 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.810827    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.812577 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.812559    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4173ef65-0d8d-4867-840a-0750921ecfeb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.819967 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.819945    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwz7\" (UniqueName: \"kubernetes.io/projected/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-api-access-dpwz7\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:09.827807 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:09.827783    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:10.100224 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:10.100198    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"]
Sep 16 19:10:10.102789 ip-10-0-143-139 kubenswrapper[2567]: W0916 19:10:10.102759    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4173ef65_0d8d_4867_840a_0750921ecfeb.slice/crio-1fc77bc7dd5ecdb4f51912eb06861bac24b61e5298c6b7b98b3fa0a1b3e4add5 WatchSource:0}: Error finding container 1fc77bc7dd5ecdb4f51912eb06861bac24b61e5298c6b7b98b3fa0a1b3e4add5: Status 404 returned error can't find the container with id 1fc77bc7dd5ecdb4f51912eb06861bac24b61e5298c6b7b98b3fa0a1b3e4add5
Sep 16 19:10:10.199218 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:10.199179    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" event={"ID":"4173ef65-0d8d-4867-840a-0750921ecfeb","Type":"ContainerStarted","Data":"d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d"}
Sep 16 19:10:10.199368 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:10.199248    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" event={"ID":"4173ef65-0d8d-4867-840a-0750921ecfeb","Type":"ContainerStarted","Data":"1fc77bc7dd5ecdb4f51912eb06861bac24b61e5298c6b7b98b3fa0a1b3e4add5"}
Sep 16 19:10:14.217801 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:14.217769    2567 generic.go:356] "Generic (PLEG): container finished" podID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerID="d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d" exitCode=0
Sep 16 19:10:14.218170 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:14.217830    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" event={"ID":"4173ef65-0d8d-4867-840a-0750921ecfeb","Type":"ContainerDied","Data":"d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d"}
Sep 16 19:10:15.224865 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:15.224833    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" event={"ID":"4173ef65-0d8d-4867-840a-0750921ecfeb","Type":"ContainerStarted","Data":"2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13"}
Sep 16 19:10:15.225293 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:15.224892    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" event={"ID":"4173ef65-0d8d-4867-840a-0750921ecfeb","Type":"ContainerStarted","Data":"ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802"}
Sep 16 19:10:15.225293 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:15.225075    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:15.248530 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:15.248481    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podStartSLOduration=2.387428362 podStartE2EDuration="6.248466573s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.861038211s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:15.247877323 +0000 UTC m=+1482.436324005" watchObservedRunningTime="2026-09-16 19:10:15.248466573 +0000 UTC m=+1482.436913254"
Sep 16 19:10:16.229547 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:16.229509    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:16.230935 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:16.230906    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:17.234146 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:17.234090    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:22.240158 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:22.240125    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:10:22.240719 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:22.240690    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:32.241361 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:32.241322    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:36.638192 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:36.638158    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 19:10:36.640650 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:36.640625    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 19:10:42.240869 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:42.240832    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:52.241261 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:10:52.241221    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:11:02.240738 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:02.240655    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:11:12.241387 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:12.241352    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:11:22.241288 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:22.241246    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:11:32.242004 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:32.241976    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:11:39.674247 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:39.674215    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"]
Sep 16 19:11:39.676702 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:39.674555    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" containerID="cri-o://ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802" gracePeriod=30
Sep 16 19:11:39.676702 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:39.674608    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kube-rbac-proxy" containerID="cri-o://2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13" gracePeriod=30
Sep 16 19:11:40.605918 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:40.605889    2567 generic.go:356] "Generic (PLEG): container finished" podID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerID="2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13" exitCode=2
Sep 16 19:11:40.606089 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:40.605953    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" event={"ID":"4173ef65-0d8d-4867-840a-0750921ecfeb","Type":"ContainerDied","Data":"2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13"}
Sep 16 19:11:42.234800 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:42.234750    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:11:42.241286 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:42.241259    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:11:44.218182 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.218156    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:11:44.332373 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.332282    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-cabundle-cert\") pod \"4173ef65-0d8d-4867-840a-0750921ecfeb\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") "
Sep 16 19:11:44.332373 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.332356    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-rbac-proxy-sar-config\") pod \"4173ef65-0d8d-4867-840a-0750921ecfeb\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") "
Sep 16 19:11:44.332590 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.332383    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4173ef65-0d8d-4867-840a-0750921ecfeb-kserve-provision-location\") pod \"4173ef65-0d8d-4867-840a-0750921ecfeb\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") "
Sep 16 19:11:44.332590 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.332412    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4173ef65-0d8d-4867-840a-0750921ecfeb-proxy-tls\") pod \"4173ef65-0d8d-4867-840a-0750921ecfeb\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") "
Sep 16 19:11:44.332590 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.332438    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpwz7\" (UniqueName: \"kubernetes.io/projected/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-api-access-dpwz7\") pod \"4173ef65-0d8d-4867-840a-0750921ecfeb\" (UID: \"4173ef65-0d8d-4867-840a-0750921ecfeb\") "
Sep 16 19:11:44.332753 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.332710    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4173ef65-0d8d-4867-840a-0750921ecfeb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4173ef65-0d8d-4867-840a-0750921ecfeb" (UID: "4173ef65-0d8d-4867-840a-0750921ecfeb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:44.332753 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.332733    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4173ef65-0d8d-4867-840a-0750921ecfeb" (UID: "4173ef65-0d8d-4867-840a-0750921ecfeb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:44.332753 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.332723    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4173ef65-0d8d-4867-840a-0750921ecfeb" (UID: "4173ef65-0d8d-4867-840a-0750921ecfeb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:44.334528 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.334499    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4173ef65-0d8d-4867-840a-0750921ecfeb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4173ef65-0d8d-4867-840a-0750921ecfeb" (UID: "4173ef65-0d8d-4867-840a-0750921ecfeb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:44.334653 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.334583    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-api-access-dpwz7" (OuterVolumeSpecName: "kube-api-access-dpwz7") pod "4173ef65-0d8d-4867-840a-0750921ecfeb" (UID: "4173ef65-0d8d-4867-840a-0750921ecfeb"). InnerVolumeSpecName "kube-api-access-dpwz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:44.434057 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.434017    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-cabundle-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:11:44.434057 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.434052    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:11:44.434287 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.434069    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4173ef65-0d8d-4867-840a-0750921ecfeb-kserve-provision-location\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:11:44.434287 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.434082    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4173ef65-0d8d-4867-840a-0750921ecfeb-proxy-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:11:44.434287 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.434095    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpwz7\" (UniqueName: \"kubernetes.io/projected/4173ef65-0d8d-4867-840a-0750921ecfeb-kube-api-access-dpwz7\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:11:44.625854 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.625822    2567 generic.go:356] "Generic (PLEG): container finished" podID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerID="ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802" exitCode=0
Sep 16 19:11:44.626022 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.625883    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"
Sep 16 19:11:44.626022 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.625890    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" event={"ID":"4173ef65-0d8d-4867-840a-0750921ecfeb","Type":"ContainerDied","Data":"ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802"}
Sep 16 19:11:44.626022 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.625939    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv" event={"ID":"4173ef65-0d8d-4867-840a-0750921ecfeb","Type":"ContainerDied","Data":"1fc77bc7dd5ecdb4f51912eb06861bac24b61e5298c6b7b98b3fa0a1b3e4add5"}
Sep 16 19:11:44.626022 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.625964    2567 scope.go:117] "RemoveContainer" containerID="2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13"
Sep 16 19:11:44.634906 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.634886    2567 scope.go:117] "RemoveContainer" containerID="ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802"
Sep 16 19:11:44.643036 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.643006    2567 scope.go:117] "RemoveContainer" containerID="d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d"
Sep 16 19:11:44.651761 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.651728    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"]
Sep 16 19:11:44.652055 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.652037    2567 scope.go:117] "RemoveContainer" containerID="2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13"
Sep 16 19:11:44.652404 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:11:44.652384    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13\": container with ID starting with 2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13 not found: ID does not exist" containerID="2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13"
Sep 16 19:11:44.652491 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.652413    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13"} err="failed to get container status \"2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13\": rpc error: code = NotFound desc = could not find container \"2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13\": container with ID starting with 2025dd450ffd99c86ceb0c3d7b223af3aa261ab6d3a682a3b263f0834e94ec13 not found: ID does not exist"
Sep 16 19:11:44.652491 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.652431    2567 scope.go:117] "RemoveContainer" containerID="ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802"
Sep 16 19:11:44.652754 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:11:44.652723    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802\": container with ID starting with ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802 not found: ID does not exist" containerID="ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802"
Sep 16 19:11:44.652825 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.652765    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802"} err="failed to get container status \"ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802\": rpc error: code = NotFound desc = could not find container \"ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802\": container with ID starting with ee6714c32c2ab461dc7977c2f690cd3ad311ff0d29487b3d02b9232bfccbc802 not found: ID does not exist"
Sep 16 19:11:44.652825 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.652789    2567 scope.go:117] "RemoveContainer" containerID="d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d"
Sep 16 19:11:44.653208 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:11:44.653176    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d\": container with ID starting with d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d not found: ID does not exist" containerID="d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d"
Sep 16 19:11:44.653356 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.653217    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d"} err="failed to get container status \"d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d\": rpc error: code = NotFound desc = could not find container \"d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d\": container with ID starting with d86bfca492feef93f465c5c49fe81c6ba5540bdfd414a08b33bba609bc27784d not found: ID does not exist"
Sep 16 19:11:44.655914 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:44.655890    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-kz9tv"]
Sep 16 19:11:45.333720 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:11:45.333690    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" path="/var/lib/kubelet/pods/4173ef65-0d8d-4867-840a-0750921ecfeb/volumes"
Sep 16 19:13:51.008091 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.008054    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zrdnl/must-gather-shtm4"]
Sep 16 19:13:51.008568 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.008419    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kube-rbac-proxy"
Sep 16 19:13:51.008568 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.008432    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kube-rbac-proxy"
Sep 16 19:13:51.008568 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.008459    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="storage-initializer"
Sep 16 19:13:51.008568 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.008465    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="storage-initializer"
Sep 16 19:13:51.008568 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.008471    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container"
Sep 16 19:13:51.008568 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.008477    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container"
Sep 16 19:13:51.008568 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.008536    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kserve-container"
Sep 16 19:13:51.008568 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.008544    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4173ef65-0d8d-4867-840a-0750921ecfeb" containerName="kube-rbac-proxy"
Sep 16 19:13:51.011462 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.011447    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:13:51.014786 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.014750    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zrdnl\"/\"kube-root-ca.crt\""
Sep 16 19:13:51.015187 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.014871    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zrdnl\"/\"openshift-service-ca.crt\""
Sep 16 19:13:51.016643 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.016620    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zrdnl\"/\"default-dockercfg-5sm27\""
Sep 16 19:13:51.023536 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.023511    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zrdnl/must-gather-shtm4"]
Sep 16 19:13:51.126683 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.126639    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkrzr\" (UniqueName: \"kubernetes.io/projected/c109d0f0-1f6c-450f-80d2-3128111980a5-kube-api-access-qkrzr\") pod \"must-gather-shtm4\" (UID: \"c109d0f0-1f6c-450f-80d2-3128111980a5\") " pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:13:51.126865 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.126696    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c109d0f0-1f6c-450f-80d2-3128111980a5-must-gather-output\") pod \"must-gather-shtm4\" (UID: \"c109d0f0-1f6c-450f-80d2-3128111980a5\") " pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:13:51.227508 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.227465    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c109d0f0-1f6c-450f-80d2-3128111980a5-must-gather-output\") pod \"must-gather-shtm4\" (UID: \"c109d0f0-1f6c-450f-80d2-3128111980a5\") " pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:13:51.227682 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.227558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkrzr\" (UniqueName: \"kubernetes.io/projected/c109d0f0-1f6c-450f-80d2-3128111980a5-kube-api-access-qkrzr\") pod \"must-gather-shtm4\" (UID: \"c109d0f0-1f6c-450f-80d2-3128111980a5\") " pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:13:51.227850 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.227826    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c109d0f0-1f6c-450f-80d2-3128111980a5-must-gather-output\") pod \"must-gather-shtm4\" (UID: \"c109d0f0-1f6c-450f-80d2-3128111980a5\") " pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:13:51.236249 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.236226    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkrzr\" (UniqueName: \"kubernetes.io/projected/c109d0f0-1f6c-450f-80d2-3128111980a5-kube-api-access-qkrzr\") pod \"must-gather-shtm4\" (UID: \"c109d0f0-1f6c-450f-80d2-3128111980a5\") " pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:13:51.340977 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.340901    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:13:51.617539 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.617511    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zrdnl/must-gather-shtm4"]
Sep 16 19:13:51.623080 ip-10-0-143-139 kubenswrapper[2567]: W0916 19:13:51.623052    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc109d0f0_1f6c_450f_80d2_3128111980a5.slice/crio-feac55639d4c889bfcb56183e15973947c5009af51600b1745fe151ec67a10a9 WatchSource:0}: Error finding container feac55639d4c889bfcb56183e15973947c5009af51600b1745fe151ec67a10a9: Status 404 returned error can't find the container with id feac55639d4c889bfcb56183e15973947c5009af51600b1745fe151ec67a10a9
Sep 16 19:13:51.624924 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:51.624906    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:13:52.195684 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:52.195639    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrdnl/must-gather-shtm4" event={"ID":"c109d0f0-1f6c-450f-80d2-3128111980a5","Type":"ContainerStarted","Data":"feac55639d4c889bfcb56183e15973947c5009af51600b1745fe151ec67a10a9"}
Sep 16 19:13:58.227674 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:58.227634    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrdnl/must-gather-shtm4" event={"ID":"c109d0f0-1f6c-450f-80d2-3128111980a5","Type":"ContainerStarted","Data":"d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4"}
Sep 16 19:13:58.228180 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:58.227687    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrdnl/must-gather-shtm4" event={"ID":"c109d0f0-1f6c-450f-80d2-3128111980a5","Type":"ContainerStarted","Data":"efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f"}
Sep 16 19:13:58.248755 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:13:58.248695    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zrdnl/must-gather-shtm4" podStartSLOduration=2.402147613 podStartE2EDuration="8.248679628s" totalImagesPullingTime="5.846532015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:58.245895114 +0000 UTC m=+1705.434341796" watchObservedRunningTime="2026-09-16 19:13:58.248679628 +0000 UTC m=+1705.437126309"
Sep 16 19:14:21.337003 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:21.336912    2567 generic.go:356] "Generic (PLEG): container finished" podID="c109d0f0-1f6c-450f-80d2-3128111980a5" containerID="efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f" exitCode=0
Sep 16 19:14:21.337003 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:21.336983    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrdnl/must-gather-shtm4" event={"ID":"c109d0f0-1f6c-450f-80d2-3128111980a5","Type":"ContainerDied","Data":"efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f"}
Sep 16 19:14:21.337464 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:21.337312    2567 scope.go:117] "RemoveContainer" containerID="efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f"
Sep 16 19:14:21.988980 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:21.988954    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zrdnl_must-gather-shtm4_c109d0f0-1f6c-450f-80d2-3128111980a5/gather/0.log"
Sep 16 19:14:22.599407 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.599369    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-68wb5/must-gather-hjq6q"]
Sep 16 19:14:22.602059 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.602040    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68wb5/must-gather-hjq6q"
Sep 16 19:14:22.605625 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.605602    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68wb5\"/\"openshift-service-ca.crt\""
Sep 16 19:14:22.605762 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.605608    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-68wb5\"/\"default-dockercfg-fd5h5\""
Sep 16 19:14:22.606950 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.606926    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68wb5\"/\"kube-root-ca.crt\""
Sep 16 19:14:22.624441 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.624413    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68wb5/must-gather-hjq6q"]
Sep 16 19:14:22.721737 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.721700    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd7s8\" (UniqueName: \"kubernetes.io/projected/f683e5e0-7532-4091-ba34-99704e2d0c0c-kube-api-access-cd7s8\") pod \"must-gather-hjq6q\" (UID: \"f683e5e0-7532-4091-ba34-99704e2d0c0c\") " pod="openshift-must-gather-68wb5/must-gather-hjq6q"
Sep 16 19:14:22.721904 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.721807    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f683e5e0-7532-4091-ba34-99704e2d0c0c-must-gather-output\") pod \"must-gather-hjq6q\" (UID: \"f683e5e0-7532-4091-ba34-99704e2d0c0c\") " pod="openshift-must-gather-68wb5/must-gather-hjq6q"
Sep 16 19:14:22.822492 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.822460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f683e5e0-7532-4091-ba34-99704e2d0c0c-must-gather-output\") pod \"must-gather-hjq6q\" (UID: \"f683e5e0-7532-4091-ba34-99704e2d0c0c\") " pod="openshift-must-gather-68wb5/must-gather-hjq6q"
Sep 16 19:14:22.822690 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.822523    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd7s8\" (UniqueName: \"kubernetes.io/projected/f683e5e0-7532-4091-ba34-99704e2d0c0c-kube-api-access-cd7s8\") pod \"must-gather-hjq6q\" (UID: \"f683e5e0-7532-4091-ba34-99704e2d0c0c\") " pod="openshift-must-gather-68wb5/must-gather-hjq6q"
Sep 16 19:14:22.822847 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.822825    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f683e5e0-7532-4091-ba34-99704e2d0c0c-must-gather-output\") pod \"must-gather-hjq6q\" (UID: \"f683e5e0-7532-4091-ba34-99704e2d0c0c\") " pod="openshift-must-gather-68wb5/must-gather-hjq6q"
Sep 16 19:14:22.831676 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.831651    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd7s8\" (UniqueName: \"kubernetes.io/projected/f683e5e0-7532-4091-ba34-99704e2d0c0c-kube-api-access-cd7s8\") pod \"must-gather-hjq6q\" (UID: \"f683e5e0-7532-4091-ba34-99704e2d0c0c\") " pod="openshift-must-gather-68wb5/must-gather-hjq6q"
Sep 16 19:14:22.911572 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:22.911546    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68wb5/must-gather-hjq6q"
Sep 16 19:14:23.190159 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:23.190062    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68wb5/must-gather-hjq6q"]
Sep 16 19:14:23.193217 ip-10-0-143-139 kubenswrapper[2567]: W0916 19:14:23.193175    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf683e5e0_7532_4091_ba34_99704e2d0c0c.slice/crio-cd39bea479be10c03d30498a6f8aa0cadcbec5ccad0dae3b4d2d4f966ffa0cf1 WatchSource:0}: Error finding container cd39bea479be10c03d30498a6f8aa0cadcbec5ccad0dae3b4d2d4f966ffa0cf1: Status 404 returned error can't find the container with id cd39bea479be10c03d30498a6f8aa0cadcbec5ccad0dae3b4d2d4f966ffa0cf1
Sep 16 19:14:23.346999 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:23.346966    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68wb5/must-gather-hjq6q" event={"ID":"f683e5e0-7532-4091-ba34-99704e2d0c0c","Type":"ContainerStarted","Data":"cd39bea479be10c03d30498a6f8aa0cadcbec5ccad0dae3b4d2d4f966ffa0cf1"}
Sep 16 19:14:24.355817 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:24.355779    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68wb5/must-gather-hjq6q" event={"ID":"f683e5e0-7532-4091-ba34-99704e2d0c0c","Type":"ContainerStarted","Data":"057be634c9a9003bb5058c82f4699a1b8ab54152670917b6198471ceed5cdbc4"}
Sep 16 19:14:24.356208 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:24.355832    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68wb5/must-gather-hjq6q" event={"ID":"f683e5e0-7532-4091-ba34-99704e2d0c0c","Type":"ContainerStarted","Data":"7f3901e11d837cb37006c2c0e3ef8c4c7d80bd858feeb3313fd1646ffc62d814"}
Sep 16 19:14:24.380254 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:24.380192    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-68wb5/must-gather-hjq6q" podStartSLOduration=1.710122914 podStartE2EDuration="2.380170867s" totalImagesPullingTime="670.047953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:24.376554464 +0000 UTC m=+1731.565001149" watchObservedRunningTime="2026-09-16 19:14:24.380170867 +0000 UTC m=+1731.568617551"
Sep 16 19:14:25.300939 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:25.300901    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8g5df_a2104af8-40ea-48de-8403-6c112903ca6d/global-pull-secret-syncer/0.log"
Sep 16 19:14:25.469804 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:25.469764    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wqzk6_7808bd97-d8b5-43a5-a11f-37dbbf196633/konnectivity-agent/0.log"
Sep 16 19:14:25.538582 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:25.538549    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-139.ec2.internal_cd53a7a796c4569b9de0a17cd4101459/haproxy/0.log"
Sep 16 19:14:27.427474 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:27.427437    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-zrdnl/must-gather-shtm4"]
Sep 16 19:14:27.428407 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:27.428372    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-zrdnl/must-gather-shtm4" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" containerName="copy" containerID="cri-o://d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4" gracePeriod=2
Sep 16 19:14:27.432004 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:27.431980    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-zrdnl/must-gather-shtm4"]
Sep 16 19:14:27.433521 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:14:27.433490    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-shtm4\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zrdnl\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:14:27.795200 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:27.794683    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zrdnl_must-gather-shtm4_c109d0f0-1f6c-450f-80d2-3128111980a5/copy/0.log"
Sep 16 19:14:27.801132 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:27.799325    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:14:27.802350 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:14:27.802310    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-shtm4\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zrdnl\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:14:27.891152 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:27.890412    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c109d0f0-1f6c-450f-80d2-3128111980a5-must-gather-output\") pod \"c109d0f0-1f6c-450f-80d2-3128111980a5\" (UID: \"c109d0f0-1f6c-450f-80d2-3128111980a5\") "
Sep 16 19:14:27.891152 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:27.890543    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkrzr\" (UniqueName: \"kubernetes.io/projected/c109d0f0-1f6c-450f-80d2-3128111980a5-kube-api-access-qkrzr\") pod \"c109d0f0-1f6c-450f-80d2-3128111980a5\" (UID: \"c109d0f0-1f6c-450f-80d2-3128111980a5\") "
Sep 16 19:14:27.894292 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:27.893085    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c109d0f0-1f6c-450f-80d2-3128111980a5-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "c109d0f0-1f6c-450f-80d2-3128111980a5" (UID: "c109d0f0-1f6c-450f-80d2-3128111980a5"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:27.894925 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:27.894902    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c109d0f0-1f6c-450f-80d2-3128111980a5-kube-api-access-qkrzr" (OuterVolumeSpecName: "kube-api-access-qkrzr") pod "c109d0f0-1f6c-450f-80d2-3128111980a5" (UID: "c109d0f0-1f6c-450f-80d2-3128111980a5"). InnerVolumeSpecName "kube-api-access-qkrzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:28.002240 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.002202    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkrzr\" (UniqueName: \"kubernetes.io/projected/c109d0f0-1f6c-450f-80d2-3128111980a5-kube-api-access-qkrzr\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:14:28.002240 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.002239    2567 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c109d0f0-1f6c-450f-80d2-3128111980a5-must-gather-output\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Sep 16 19:14:28.379767 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.379731    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zrdnl_must-gather-shtm4_c109d0f0-1f6c-450f-80d2-3128111980a5/copy/0.log"
Sep 16 19:14:28.380204 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.380171    2567 generic.go:356] "Generic (PLEG): container finished" podID="c109d0f0-1f6c-450f-80d2-3128111980a5" containerID="d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4" exitCode=143
Sep 16 19:14:28.380311 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.380276    2567 scope.go:117] "RemoveContainer" containerID="d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4"
Sep 16 19:14:28.380470 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.380453    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:14:28.394624 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:14:28.394589    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-shtm4\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zrdnl\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:14:28.416225 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.416202    2567 scope.go:117] "RemoveContainer" containerID="efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f"
Sep 16 19:14:28.416705 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:14:28.416519    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-shtm4\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zrdnl\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" pod="openshift-must-gather-zrdnl/must-gather-shtm4"
Sep 16 19:14:28.443455 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.443428    2567 scope.go:117] "RemoveContainer" containerID="d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4"
Sep 16 19:14:28.444333 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:14:28.444301    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4\": container with ID starting with d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4 not found: ID does not exist" containerID="d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4"
Sep 16 19:14:28.444499 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.444472    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4"} err="failed to get container status \"d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4\": rpc error: code = NotFound desc = could not find container \"d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4\": container with ID starting with d9e2f0e0ea40a176813971116661eb1f0d895e5afaaf0d6f430a65e317c660f4 not found: ID does not exist"
Sep 16 19:14:28.444611 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.444600    2567 scope.go:117] "RemoveContainer" containerID="efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f"
Sep 16 19:14:28.445099 ip-10-0-143-139 kubenswrapper[2567]: E0916 19:14:28.445077    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f\": container with ID starting with efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f not found: ID does not exist" containerID="efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f"
Sep 16 19:14:28.445288 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.445265    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f"} err="failed to get container status \"efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f\": rpc error: code = NotFound desc = could not find container \"efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f\": container with ID starting with efed1918df9340dc02642e5b82133484751aa8d19023b1afac9926f4ad9cde9f not found: ID does not exist"
Sep 16 19:14:28.815389 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.815308    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e96dfc47-4e0f-4a77-8e65-1282b08a22ee/alertmanager/0.log"
Sep 16 19:14:28.838725 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.838682    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e96dfc47-4e0f-4a77-8e65-1282b08a22ee/config-reloader/0.log"
Sep 16 19:14:28.865626 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.865599    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e96dfc47-4e0f-4a77-8e65-1282b08a22ee/kube-rbac-proxy-web/0.log"
Sep 16 19:14:28.893387 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.893363    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e96dfc47-4e0f-4a77-8e65-1282b08a22ee/kube-rbac-proxy/0.log"
Sep 16 19:14:28.926884 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.926858    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e96dfc47-4e0f-4a77-8e65-1282b08a22ee/kube-rbac-proxy-metric/0.log"
Sep 16 19:14:28.968590 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.968551    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e96dfc47-4e0f-4a77-8e65-1282b08a22ee/prom-label-proxy/0.log"
Sep 16 19:14:28.998370 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:28.998341    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e96dfc47-4e0f-4a77-8e65-1282b08a22ee/init-config-reloader/0.log"
Sep 16 19:14:29.081682 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.081599    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-5qgmh_49e3d9c7-c939-45a4-acc1-940afde69201/kube-state-metrics/0.log"
Sep 16 19:14:29.107651 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.107620    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-5qgmh_49e3d9c7-c939-45a4-acc1-940afde69201/kube-rbac-proxy-main/0.log"
Sep 16 19:14:29.139205 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.139174    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-5qgmh_49e3d9c7-c939-45a4-acc1-940afde69201/kube-rbac-proxy-self/0.log"
Sep 16 19:14:29.197823 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.197786    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-56b5948bdb-qxwmw_541e723c-a0a6-447f-b16f-72359bc7c382/metrics-server/0.log"
Sep 16 19:14:29.280345 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.280313    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5b246_d8c25ba8-9217-4dcc-b645-00ca89c6641e/node-exporter/0.log"
Sep 16 19:14:29.305969 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.305923    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5b246_d8c25ba8-9217-4dcc-b645-00ca89c6641e/kube-rbac-proxy/0.log"
Sep 16 19:14:29.351624 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.351451    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5b246_d8c25ba8-9217-4dcc-b645-00ca89c6641e/init-textfile/0.log"
Sep 16 19:14:29.357132 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.354491    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" path="/var/lib/kubelet/pods/c109d0f0-1f6c-450f-80d2-3128111980a5/volumes"
Sep 16 19:14:29.610667 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.610581    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-8fwdr_6619a7cc-8fc6-43ce-8a48-7a1ac0880821/kube-rbac-proxy-main/0.log"
Sep 16 19:14:29.649243 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.649212    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-8fwdr_6619a7cc-8fc6-43ce-8a48-7a1ac0880821/kube-rbac-proxy-self/0.log"
Sep 16 19:14:29.683873 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.683841    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-8fwdr_6619a7cc-8fc6-43ce-8a48-7a1ac0880821/openshift-state-metrics/0.log"
Sep 16 19:14:29.744834 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.744798    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be31a6-f749-46c3-99d5-97c37a045d77/prometheus/0.log"
Sep 16 19:14:29.795419 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.795378    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be31a6-f749-46c3-99d5-97c37a045d77/config-reloader/0.log"
Sep 16 19:14:29.843801 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.843771    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be31a6-f749-46c3-99d5-97c37a045d77/thanos-sidecar/0.log"
Sep 16 19:14:29.878514 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.878480    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be31a6-f749-46c3-99d5-97c37a045d77/kube-rbac-proxy-web/0.log"
Sep 16 19:14:29.902514 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.902487    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be31a6-f749-46c3-99d5-97c37a045d77/kube-rbac-proxy/0.log"
Sep 16 19:14:29.927499 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.927466    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be31a6-f749-46c3-99d5-97c37a045d77/kube-rbac-proxy-thanos/0.log"
Sep 16 19:14:29.954227 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:29.954197    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be31a6-f749-46c3-99d5-97c37a045d77/init-config-reloader/0.log"
Sep 16 19:14:30.048949 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:30.048905    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-hgmkg_85b5a4ec-d7ab-4ad9-9cf6-e1b3c0a271c3/prometheus-operator-admission-webhook/0.log"
Sep 16 19:14:30.082239 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:30.082200    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bddd84c9-gvjhh_a1ea8c55-34c4-4860-b8a3-c601090ef138/telemeter-client/0.log"
Sep 16 19:14:30.105676 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:30.105650    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bddd84c9-gvjhh_a1ea8c55-34c4-4860-b8a3-c601090ef138/reload/0.log"
Sep 16 19:14:30.138679 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:30.138606    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bddd84c9-gvjhh_a1ea8c55-34c4-4860-b8a3-c601090ef138/kube-rbac-proxy/0.log"
Sep 16 19:14:31.767093 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:31.767017    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-4gg9x_7d95f7bc-6499-4c74-9c04-41a9b501f6ba/networking-console-plugin/0.log"
Sep 16 19:14:32.349098 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:32.348900    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/1.log"
Sep 16 19:14:32.356452 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:32.356428    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7khsl_007217df-062a-4583-8425-a5f6df57af08/console-operator/2.log"
Sep 16 19:14:32.851360 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:32.851333    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f4dc4d8bd-hhzf7_423fe583-eac8-41b6-b7a1-b5a37d421752/console/0.log"
Sep 16 19:14:33.395388 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.395362    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-l7d59_3c6ef1b4-3a09-40c2-bc93-d99b1e8962b8/volume-data-source-validator/0.log"
Sep 16 19:14:33.869848 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.869808    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"]
Sep 16 19:14:33.870380 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.870354    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" containerName="gather"
Sep 16 19:14:33.870380 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.870379    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" containerName="gather"
Sep 16 19:14:33.870536 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.870412    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" containerName="copy"
Sep 16 19:14:33.870536 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.870421    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" containerName="copy"
Sep 16 19:14:33.870536 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.870498    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" containerName="copy"
Sep 16 19:14:33.870536 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.870509    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c109d0f0-1f6c-450f-80d2-3128111980a5" containerName="gather"
Sep 16 19:14:33.873089 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.873062    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:33.886085 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.886061    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"]
Sep 16 19:14:33.961195 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.961104    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-podres\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:33.961376 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.961204    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-sys\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:33.961376 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.961229    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-proc\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:33.961376 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.961268    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgfd\" (UniqueName: \"kubernetes.io/projected/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-kube-api-access-drgfd\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:33.961376 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:33.961349    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-lib-modules\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.062911 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.062866    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-lib-modules\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.063098 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.062970    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-podres\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.063098 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.062997    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-sys\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.063098 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.063022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-proc\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.063098 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.063063    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgfd\" (UniqueName: \"kubernetes.io/projected/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-kube-api-access-drgfd\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.063453 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.063430    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-sys\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.063531 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.063465    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-proc\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.063531 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.063472    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-podres\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.063531 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.063469    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-lib-modules\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.073256 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.073228    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgfd\" (UniqueName: \"kubernetes.io/projected/3da6c6c7-d8b5-4332-8ac4-48958b7a2195-kube-api-access-drgfd\") pod \"perf-node-gather-daemonset-gsl5c\" (UID: \"3da6c6c7-d8b5-4332-8ac4-48958b7a2195\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.185075 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.185000    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:34.429013 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.428986    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mf48l_3b545178-c81e-4173-9675-50aaa359f497/dns/0.log"
Sep 16 19:14:34.454381 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.454285    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mf48l_3b545178-c81e-4173-9675-50aaa359f497/kube-rbac-proxy/0.log"
Sep 16 19:14:34.484792 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.484770    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"]
Sep 16 19:14:34.488091 ip-10-0-143-139 kubenswrapper[2567]: W0916 19:14:34.488063    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3da6c6c7_d8b5_4332_8ac4_48958b7a2195.slice/crio-b7a90319fdf29853985992be6da1221c354a9c0753eb19b48e6ce2a59a6096b6 WatchSource:0}: Error finding container b7a90319fdf29853985992be6da1221c354a9c0753eb19b48e6ce2a59a6096b6: Status 404 returned error can't find the container with id b7a90319fdf29853985992be6da1221c354a9c0753eb19b48e6ce2a59a6096b6
Sep 16 19:14:34.663616 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:34.663591    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bhsmz_ea444e57-cd1a-4620-a3aa-df7e8348c481/dns-node-resolver/0.log"
Sep 16 19:14:35.258241 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:35.258208    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zbkkp_e69d4209-41fd-4daf-8d4b-e9fd7f377777/node-ca/0.log"
Sep 16 19:14:35.427750 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:35.427706    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c" event={"ID":"3da6c6c7-d8b5-4332-8ac4-48958b7a2195","Type":"ContainerStarted","Data":"2c2d70519f3b5bf040c90857ca383dcb82989dbb1eab07ca42303eb6b328e5cf"}
Sep 16 19:14:35.427935 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:35.427776    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c" event={"ID":"3da6c6c7-d8b5-4332-8ac4-48958b7a2195","Type":"ContainerStarted","Data":"b7a90319fdf29853985992be6da1221c354a9c0753eb19b48e6ce2a59a6096b6"}
Sep 16 19:14:35.427935 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:35.427905    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:36.370419 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:36.370377    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gv2gr_9dc205a3-9cc1-4f77-9b00-8df61cf69ba4/serve-healthcheck-canary/0.log"
Sep 16 19:14:36.754567 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:36.754453    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-2jhcw_81bd4b21-6b45-4504-b47f-a6201451b987/insights-operator/0.log"
Sep 16 19:14:36.755058 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:36.755037    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-2jhcw_81bd4b21-6b45-4504-b47f-a6201451b987/insights-operator/1.log"
Sep 16 19:14:36.915656 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:36.915632    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkl5v_74280712-1f17-494c-a01e-e0c17d385c96/kube-rbac-proxy/0.log"
Sep 16 19:14:36.935440 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:36.935419    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkl5v_74280712-1f17-494c-a01e-e0c17d385c96/exporter/0.log"
Sep 16 19:14:36.955955 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:36.955921    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkl5v_74280712-1f17-494c-a01e-e0c17d385c96/extractor/0.log"
Sep 16 19:14:39.429661 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:39.429636    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vstz2_fa85cfaf-4aa1-4861-afda-64c62b1a566a/s3-init/0.log"
Sep 16 19:14:39.446407 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:39.446382    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vstz2_fa85cfaf-4aa1-4861-afda-64c62b1a566a/download-hf-model/0.log"
Sep 16 19:14:39.472013 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:39.471983    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-rtp9z_3bbcb5f1-2da0-42c5-97c8-ab3e8f7766f2/seaweedfs/0.log"
Sep 16 19:14:41.446974 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:41.446948    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c"
Sep 16 19:14:41.472594 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:41.472536    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-gsl5c" podStartSLOduration=8.472515605 podStartE2EDuration="8.472515605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:35.461410259 +0000 UTC m=+1742.649856941" watchObservedRunningTime="2026-09-16 19:14:41.472515605 +0000 UTC m=+1748.660962288"
Sep 16 19:14:46.049537 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:46.049469    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfxbv_9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:14:46.110491 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:46.110460    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfxbv_9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298/egress-router-binary-copy/0.log"
Sep 16 19:14:46.146001 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:46.145978    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfxbv_9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298/cni-plugins/0.log"
Sep 16 19:14:46.188122 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:46.188075    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfxbv_9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298/bond-cni-plugin/0.log"
Sep 16 19:14:46.236101 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:46.236073    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfxbv_9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298/routeoverride-cni/0.log"
Sep 16 19:14:46.265735 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:46.265698    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfxbv_9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298/whereabouts-cni-bincopy/0.log"
Sep 16 19:14:46.308148 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:46.308063    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfxbv_9f6613b2-b8d0-48a1-bfd2-3eaeca2ae298/whereabouts-cni/0.log"
Sep 16 19:14:46.584417 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:46.584341    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p2z8b_8856e3f1-9640-4dc7-b3fd-6936fc04f239/kube-multus/0.log"
Sep 16 19:14:46.689922 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:46.689890    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fsv9w_e7fa29a5-39f2-485f-b1fe-268334ec4935/network-metrics-daemon/0.log"
Sep 16 19:14:46.718358 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:46.718325    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fsv9w_e7fa29a5-39f2-485f-b1fe-268334ec4935/kube-rbac-proxy/0.log"
Sep 16 19:14:47.741538 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:47.741501    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69rtg_6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c/ovn-controller/0.log"
Sep 16 19:14:47.773771 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:47.773735    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69rtg_6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c/ovn-acl-logging/0.log"
Sep 16 19:14:47.811038 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:47.811012    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69rtg_6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c/kube-rbac-proxy-node/0.log"
Sep 16 19:14:47.830736 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:47.830714    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69rtg_6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:14:47.851183 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:47.851150    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69rtg_6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c/northd/0.log"
Sep 16 19:14:47.870960 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:47.870937    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69rtg_6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c/nbdb/0.log"
Sep 16 19:14:47.893044 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:47.893018    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69rtg_6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c/sbdb/0.log"
Sep 16 19:14:47.938242 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:47.938188    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69rtg_6ef3304e-ccf6-4f23-b0f6-6b9aab4cdb2c/ovnkube-controller/0.log"
Sep 16 19:14:49.339913 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:49.339882    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-g8wrg_a17239c3-07cb-400b-ac0c-36941de91c01/network-check-target-container/0.log"
Sep 16 19:14:50.312579 ip-10-0-143-139 kubenswrapper[2567]: I0916 19:14:50.312555    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-fbnt4_60a41c27-f58c-4070-b3bb-1172234dba44/iptables-alerter/0.log"