Sep 18 19:12:18.871167 ip-10-0-138-38 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 19:12:18.871176 ip-10-0-138-38 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 19:12:18.871183 ip-10-0-138-38 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 19:12:18.871326 ip-10-0-138-38 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 19:12:30.207325 ip-10-0-138-38 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 19:12:30.207343 ip-10-0-138-38 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7ebad511a9ba478d94ef1b748cad1b9f --
Sep 18 19:14:36.114501 ip-10-0-138-38 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:14:39.477437 ip-10-0-138-38 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:39.477437 ip-10-0-138-38 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:14:39.477437 ip-10-0-138-38 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:39.477437 ip-10-0-138-38 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:14:39.477437 ip-10-0-138-38 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:39.478336 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.477521    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:14:39.481734 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481708    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:39.481734 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481729    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:39.481734 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481732    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:39.481734 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481736    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:39.481734 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481739    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:39.481734 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481742    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481745    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481748    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481751    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481754    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481756    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481760    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481763    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481765    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481768    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481771    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481774    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481777    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481779    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481782    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481784    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481787    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481790    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481792    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481795    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:39.481970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481799    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481801    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481804    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481807    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481810    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481812    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481814    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481817    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481820    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481822    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481825    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481829    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481831    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481835    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481838    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481841    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481843    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481846    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481849    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481852    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:39.482463 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481854    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481857    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481859    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481862    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481864    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481873    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481876    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481879    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481881    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481884    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481886    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481891    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481895    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481898    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481901    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481904    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481906    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481909    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481912    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:39.483005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481915    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481917    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481921    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481925    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481928    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481931    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481934    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481938    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481941    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481944    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481948    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481951    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481954    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481956    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481960    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481962    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481965    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481968    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481970    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:39.483491 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481975    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481978    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481982    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481986    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481988    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481991    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481993    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481996    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.481999    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482001    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482004    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482006    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482009    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482011    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482014    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482016    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482714    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482721    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482724    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:39.483958 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482727    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482729    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482733    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482735    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482738    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482741    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482743    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482746    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482749    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482752    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482755    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482758    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482760    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482763    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482767    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482770    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482773    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482776    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482779    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:39.484442 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482783    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482787    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482791    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482794    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482797    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482800    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482804    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482806    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482809    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482811    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482814    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482817    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482820    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482822    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482825    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482828    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482831    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482833    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482836    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482838    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:39.484913 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482841    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482844    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482847    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482849    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482852    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482854    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482857    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482860    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482863    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482866    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482869    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482871    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482874    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482877    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482880    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482882    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482885    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482888    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482890    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482893    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:39.485425 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482896    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482898    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482910    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482914    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482916    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482919    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482923    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482927    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482930    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482933    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482935    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482938    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482941    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482943    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482946    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482950    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482954    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482957    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482960    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:39.485946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482963    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482966    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482970    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482973    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482975    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482978    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482981    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482983    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482986    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482989    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482991    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482993    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482996    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.482999    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483001    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483004    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483006    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483009    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483440    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483447    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:39.486441 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483451    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483454    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483457    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483460    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483462    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483465    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483468    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483470    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483473    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483475    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483478    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483483    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483485    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483489    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483494    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483497    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483501    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483504    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483507    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483509    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:39.486946 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483512    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483515    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483518    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483521    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483524    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483526    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483529    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483532    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483535    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483538    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483542    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483546    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483550    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483553    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483555    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483558    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483560    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483563    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483565    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:39.487454 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483570    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483573    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483576    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483579    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483583    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483586    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483588    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483591    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483595    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483597    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483599    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483602    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483604    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483608    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483611    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483613    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483616    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483618    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483621    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483623    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:39.487916 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483626    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483628    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483631    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483633    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483636    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483639    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483642    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483645    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483647    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483650    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483653    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483655    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483657    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483660    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483663    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483665    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483668    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483670    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483672    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483675    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:39.488414 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483677    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483681    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483683    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483686    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483689    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483691    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483693    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483696    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483699    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483701    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483704    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483707    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483710    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483712    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483715    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483717    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483720    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.483722    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483809    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483845    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:14:39.488896 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483851    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483856    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483860    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483863    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483868    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483872    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483875    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483878    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483881    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483885    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483888    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483891    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483894    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483897    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483900    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483903    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483907    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483910    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483913    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483916    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483919    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483923    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483926    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:14:39.489407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483929    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483932    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483935    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483938    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483941    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483944    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483947    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483951    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483954    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483957    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483960    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483963    2559 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483966    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483977    2559 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483980    2559 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483983    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483986    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483989    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483992    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483995    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.483998    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484001    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484004    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484007    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484011    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:14:39.489963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484014    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484017    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484020    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484023    2559 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484028    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484031    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484035    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484038    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484041    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484044    2559 flags.go:64] FLAG: --help="false"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484047    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484052    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484055    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484058    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484061    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484065    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484067    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484070    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484073    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484076    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484079    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484082    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484085    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484087    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:14:39.490610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484090    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484093    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484096    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484099    2559 flags.go:64] FLAG: --lock-file=""
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484102    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484105    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484108    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484113    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484116    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484119    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484122    2559 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484125    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484129    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484132    2559 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484135    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484139    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484142    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484146    2559 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484149    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484153    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484156    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484159    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484162    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484165    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484168    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:14:39.491260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484175    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484178    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484181    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484183    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484186    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484191    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484194    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484197    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484214    2559 flags.go:64] FLAG: --port="10250"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484217    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484220    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08b50890a1bd45c3e"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484223    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484226    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484229    2559 flags.go:64] FLAG: --register-node="true"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484232    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484235    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484239    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484242    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484244    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484248    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484253    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484256    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484259    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484261    2559 flags.go:64] FLAG: --runonce="false"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484264    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:14:39.491903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484267    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484273    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484276    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484279    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484282    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484285    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484288    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484291    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484294    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484296    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484299    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484302    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484305    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484308    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484313    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484316    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484318    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484322    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484325    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484328    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484330    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484333    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484336    2559 flags.go:64] FLAG: --v="2"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484340    2559 flags.go:64] FLAG: --version="false"
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484345    2559 flags.go:64] FLAG: --vmodule=""
Sep 18 19:14:39.492576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484348    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.484351    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484485    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484489    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484493    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484496    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484499    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484502    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484509    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484512    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484515    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484518    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484521    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484523    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484526    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484530    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484534    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484537    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484540    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484543    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:39.493188 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484545    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484548    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484551    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484554    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484556    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484559    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484561    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484564    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484566    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484569    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484572    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484575    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484578    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484580    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484585    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484587    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484591    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484595    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484598    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:39.493707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484600    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484604    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484607    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484609    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484612    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484614    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484617    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484620    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484623    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484625    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484628    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484631    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484633    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484636    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484638    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484641    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484643    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484646    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484650    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:39.494185 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484653    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484656    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484660    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484663    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484665    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484668    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484671    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484674    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484678    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484681    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484684    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484686    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484689    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484692    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484695    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484698    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484701    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484703    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484706    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484708    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:39.494707 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484711    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484714    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484716    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484719    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484722    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484724    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484727    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484729    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484732    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484734    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484737    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484740    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484743    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484745    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484748    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484750    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484753    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484755    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484758    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484760    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:39.495215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484765    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484768    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.484770    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.485524    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.492473    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.492489    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.492506    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.492511    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492567    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492572    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492576    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492579    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492582    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492585    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:39.495708 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492589    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492591    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492594    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492597    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492600    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492602    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492606    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492608    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492612    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492616    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492619    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492622    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492624    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492627    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492630    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492632    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492635    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492637    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492641    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:39.496060 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492643    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492646    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492649    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492651    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492654    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492657    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492660    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492663    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492665    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492668    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492670    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492673    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492676    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492679    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492681    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492684    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492687    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492689    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492692    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492694    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:39.496649 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492697    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492700    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492702    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492705    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492707    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492711    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492715    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492718    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492721    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492724    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492726    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492729    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492732    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492734    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492737    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492739    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492743    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492746    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492748    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:39.497172 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492751    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492754    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492756    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492759    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492761    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492764    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492767    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492769    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492772    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492774    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492777    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492779    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492782    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492785    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492788    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492790    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492793    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492795    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492798    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492800    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:39.497970 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492803    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492805    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492808    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492811    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492813    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492816    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492819    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492822    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492824    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492829    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492833    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492836    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492839    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492842    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:39.498613 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492845    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.492850    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492958    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492962    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492965    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492968    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492971    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492974    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492977    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492979    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492982    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492985    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492987    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492990    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492992    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:39.499008 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492995    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.492997    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493000    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493002    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493005    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493007    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493010    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493013    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493016    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493019    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493022    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493025    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493027    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493030    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493033    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493035    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493038    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493040    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493043    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493046    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:39.499504 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493048    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493050    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493053    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493056    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493058    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493060    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493063    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493065    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493069    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493073    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493076    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493078    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493081    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493084    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493087    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493089    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493092    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493094    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493097    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493099    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:39.500056 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493102    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493104    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493107    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493110    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493113    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493116    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493118    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493121    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493123    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493126    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493129    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493131    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493134    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493137    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493139    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493142    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493144    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493147    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493149    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:39.500563 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493152    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493155    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493158    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493161    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493164    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493166    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493169    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493171    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493174    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493176    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493179    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493182    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493184    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493186    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493189    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493192    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493195    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493211    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493214    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:39.501090 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493219    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493222    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493226    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493229    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493232    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493235    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493238    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:39.493240    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.493246    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.493338    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.496213    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.497044    2559 server.go:1019] "Starting client certificate rotation"
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.497155    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:14:39.501614 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.497219    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:14:39.502004 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.499556    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:14:39.502004 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.500968    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:14:39.519451 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.519429    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:14:39.520974 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.520959    2559 log.go:25] "Validated CRI v1 image API"
Sep 18 19:14:39.521855 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.521836    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:14:39.524232 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.524194    2559 fs.go:135] Filesystem UUIDs: map[544b6673-bc23-441b-897f-2d21ca847b58:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fe99139c-0597-4591-a11a-dc72d641b73f:/dev/nvme0n1p4]
Sep 18 19:14:39.524303 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.524231    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:14:39.528878 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.528767    2559 manager.go:216] Machine: {Timestamp:2026-09-18 19:14:39.52825989 +0000 UTC m=+1.982398059 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099118 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f2fed8c41c2a81a2a3514d4408e23 SystemUUID:ec2f2fed-8c41-c2a8-1a2a-3514d4408e23 BootID:7ebad511-a9ba-478d-94ef-1b748cad1b9f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cd:a3:ee:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cd:a3:ee:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:73:7f:b9:a2:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:14:39.528878 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.528871    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:14:39.528982 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.528970    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:14:39.530677 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.530650    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:14:39.530812 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.530679    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:14:39.531843 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.531828    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:14:39.531843 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.531844    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:14:39.531955 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.531852    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:14:39.531955 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.531864    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:14:39.531955 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.531932    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:14:39.537056 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.537044    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:14:39.537342 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.537331    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:14:39.537375 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.537368    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:14:39.537402 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.537381    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:14:39.537402 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.537397    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:14:39.537458 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.537406    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:14:39.537458 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.537415    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:14:39.537758 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.537745    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:14:39.537807 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.537767    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:14:39.539994 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.539976    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:14:39.543275 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.543261    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:14:39.543330 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.543292    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:14:39.544246 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544236    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:14:39.544285 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544251    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:14:39.544285 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544258    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:14:39.544285 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544263    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:14:39.544285 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544271    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:14:39.544285 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544276    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:14:39.544409 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544332    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:14:39.544409 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544338    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:14:39.544409 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544345    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:14:39.544409 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544351    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:14:39.544409 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544359    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:14:39.544409 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544367    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:14:39.544409 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544389    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:14:39.544409 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.544395    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:14:39.548108 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.548087    2559 server.go:1268] "Started kubelet"
Sep 18 19:14:39.548238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.548152    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:14:39.548238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.548180    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:14:39.548336 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.548287    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:14:39.549075 ip-10-0-138-38 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:14:39.549524 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.549330    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:14:39.576098 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.576076    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:14:39.576670 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.576637    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-38.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:14:39.577135 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.577092    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:14:39.577135 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.577108    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:14:39.578161 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.578113    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:14:39.580040 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.580021    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:39.581787 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.581774    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:14:39.581844 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.581803    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:39.581956 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.581899    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:39.582798 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.582625    2559 factory.go:55] Registering systemd factory
Sep 18 19:14:39.582933 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.582896    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:14:39.583041 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.582694    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:14:39.583041 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.582978    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:14:39.583041 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.582712    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:14:39.583041 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.583028    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:14:39.583041 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.583039    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:14:39.583278 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.583118    2559 factory.go:153] Registering CRI-O factory
Sep 18 19:14:39.583278 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.583131    2559 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:14:39.583278 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.583229    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:14:39.583278 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.583251    2559 factory.go:103] Registering Raw factory
Sep 18 19:14:39.583278 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.583274    2559 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:14:39.583621 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.583610    2559 manager.go:349] Starting recovery of all containers
Sep 18 19:14:39.585991 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.585966    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:14:39.586753 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.586718    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:14:39.587167 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.586081    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-38.ec2.internal.18d68037e91037e2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-38.ec2.internal,UID:ip-10-0-138-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-38.ec2.internal,},FirstTimestamp:2026-09-18 19:14:39.548061666 +0000 UTC m=+2.002199833,LastTimestamp:2026-09-18 19:14:39.548061666 +0000 UTC m=+2.002199833,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-38.ec2.internal,}"
Sep 18 19:14:39.593251 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.593232    2559 manager.go:354] Recovery completed
Sep 18 19:14:39.595392 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.595371    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ct8n4"
Sep 18 19:14:39.597773 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.597760    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:39.599658 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.599642    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:39.599743 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.599676    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:39.599743 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.599701    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:39.600296 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.600280    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:14:39.600296 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.600293    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:14:39.600406 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.600309    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:14:39.600406 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.600317    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:39.601094 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.601081    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:39.601151 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.601103    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:39.601151 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.601116    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:39.602564 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.602553    2559 policy_none.go:49] "None policy: Start"
Sep 18 19:14:39.602598 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.602569    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:14:39.602598 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.602579    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:14:39.604566 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.604555    2559 policy_none.go:47] "Start"
Sep 18 19:14:39.610805 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.610741    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-38.ec2.internal.18d68037ec23858a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-38.ec2.internal,UID:ip-10-0-138-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-38.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-38.ec2.internal,},FirstTimestamp:2026-09-18 19:14:39.599658378 +0000 UTC m=+2.053796550,LastTimestamp:2026-09-18 19:14:39.599658378 +0000 UTC m=+2.053796550,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-38.ec2.internal,}"
Sep 18 19:14:39.617589 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.617567    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ct8n4"
Sep 18 19:14:39.621112 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.621048    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-38.ec2.internal.18d68037ec240d4b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-38.ec2.internal,UID:ip-10-0-138-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-38.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-38.ec2.internal,},FirstTimestamp:2026-09-18 19:14:39.599693131 +0000 UTC m=+2.053831300,LastTimestamp:2026-09-18 19:14:39.599693131 +0000 UTC m=+2.053831300,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-38.ec2.internal,}"
Sep 18 19:14:39.637793 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.637777    2559 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:14:39.653390 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.637820    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:14:39.653390 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.637831    2559 server.go:87] "Starting device plugin registration server"
Sep 18 19:14:39.653390 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.638049    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:14:39.653390 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.638068    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:14:39.653390 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.638153    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:14:39.653390 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.638250    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:14:39.653390 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.638259    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:14:39.653390 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.638729    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:14:39.653390 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.638771    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:39.703280 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.703247    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:14:39.704425 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.704408    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:14:39.704488 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.704460    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:14:39.704531 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.704505    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:14:39.704576 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.704552    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:14:39.706870 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.706850    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:39.738562 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.738506    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:39.739547 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.739531    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:39.739617 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.739561    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:39.739617 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.739571    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:39.739617 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.739594    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.759030 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.759014    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.759084 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.759032    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-38.ec2.internal\": node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:39.806052 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.805108    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal"]
Sep 18 19:14:39.806122 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.806090    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:39.806934 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.806916    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:39.807004 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.806946    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:39.807004 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.806957    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:39.809098 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.809087    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:39.809246 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.809233    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.809287 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.809262    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:39.809866 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.809849    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:39.809866 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.809865    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:39.810016 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.809875    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:39.810016 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.809885    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:39.810016 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.809887    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:39.810016 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.809900    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:39.811952 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.811934    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.812020 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.811965    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:39.812623 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.812608    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:39.812693 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.812632    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:39.812693 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.812643    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:39.827244 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.827221    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:39.838917 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.838889    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-38.ec2.internal\" not found" node="ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.843268 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.843253    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-38.ec2.internal\" not found" node="ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.884398 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.884382    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1464c738f5162da5f46c52a035e4932a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal\" (UID: \"1464c738f5162da5f46c52a035e4932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.884476 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.884404    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/057a0ec2fb9c26d68b86efbdbfc5dfe9-config\") pod \"kube-apiserver-proxy-ip-10-0-138-38.ec2.internal\" (UID: \"057a0ec2fb9c26d68b86efbdbfc5dfe9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.884476 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.884426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1464c738f5162da5f46c52a035e4932a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal\" (UID: \"1464c738f5162da5f46c52a035e4932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.927436 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:39.927416    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:39.985413 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.985395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1464c738f5162da5f46c52a035e4932a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal\" (UID: \"1464c738f5162da5f46c52a035e4932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.985513 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.985421    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1464c738f5162da5f46c52a035e4932a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal\" (UID: \"1464c738f5162da5f46c52a035e4932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.985513 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.985440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/057a0ec2fb9c26d68b86efbdbfc5dfe9-config\") pod \"kube-apiserver-proxy-ip-10-0-138-38.ec2.internal\" (UID: \"057a0ec2fb9c26d68b86efbdbfc5dfe9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.985513 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.985494    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1464c738f5162da5f46c52a035e4932a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal\" (UID: \"1464c738f5162da5f46c52a035e4932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.985604 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.985541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/057a0ec2fb9c26d68b86efbdbfc5dfe9-config\") pod \"kube-apiserver-proxy-ip-10-0-138-38.ec2.internal\" (UID: \"057a0ec2fb9c26d68b86efbdbfc5dfe9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:39.985604 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:39.985564    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1464c738f5162da5f46c52a035e4932a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal\" (UID: \"1464c738f5162da5f46c52a035e4932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:40.028520 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:40.028501    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:40.128825 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:40.128808    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:40.142980 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.142954    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:40.146386 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.146371    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:40.229900 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:40.229870    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:40.330463 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:40.330402    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:40.430958 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:40.430929    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:40.497412 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.497385    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:14:40.497977 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.497521    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:14:40.531758 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:40.531734    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:40.547209 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.547180    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:40.623436 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.623370    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:09:39 +0000 UTC" deadline="2028-03-16 06:49:45.705092079 +0000 UTC"
Sep 18 19:14:40.623436 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.623397    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13067h35m5.081699057s"
Sep 18 19:14:40.632531 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:40.632511    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:40.664807 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.664788    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:40.667249 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.667231    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:40.667342 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.667263    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:40.667342 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.667273    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:40.732992 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:40.732964    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:40.813969 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:40.813932    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057a0ec2fb9c26d68b86efbdbfc5dfe9.slice/crio-aaa46619f1472a4c2f5b125127e869bf873561ff7d63c1ee2c5105ff92feb219 WatchSource:0}: Error finding container aaa46619f1472a4c2f5b125127e869bf873561ff7d63c1ee2c5105ff92feb219: Status 404 returned error can't find the container with id aaa46619f1472a4c2f5b125127e869bf873561ff7d63c1ee2c5105ff92feb219
Sep 18 19:14:40.814228 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:40.814185    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1464c738f5162da5f46c52a035e4932a.slice/crio-1e5f6430af43ab224eb8c526f422adee842d4b4ccf7f83ac81e4f376f9dcd41d WatchSource:0}: Error finding container 1e5f6430af43ab224eb8c526f422adee842d4b4ccf7f83ac81e4f376f9dcd41d: Status 404 returned error can't find the container with id 1e5f6430af43ab224eb8c526f422adee842d4b4ccf7f83ac81e4f376f9dcd41d
Sep 18 19:14:40.819553 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.819539    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:14:40.833856 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:40.833837    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:40.934863 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:40.934764    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-38.ec2.internal\" not found"
Sep 18 19:14:40.996787 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:40.996756    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:41.082015 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.081976    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:41.109579 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.109555    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:41.110634 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.110622    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal"
Sep 18 19:14:41.142593 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.142570    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:41.177559 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.177537    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:41.310234 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.310193    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:41.538523 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.538441    2559 apiserver.go:52] "Watching apiserver"
Sep 18 19:14:41.552064 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.552036    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:14:41.552480 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.552452    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-72js9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal","openshift-multus/network-metrics-daemon-g9cck","openshift-network-diagnostics/network-check-target-5hs2m","openshift-network-operator/iptables-alerter-hwb58","kube-system/global-pull-secret-syncer-pxfxm","openshift-dns/node-resolver-67hqf","openshift-image-registry/node-ca-xfkrz","openshift-multus/multus-87xpd","openshift-multus/multus-additional-cni-plugins-lhfb5","openshift-ovn-kubernetes/ovnkube-node-7hfmt","kube-system/konnectivity-agent-6lhsw","kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"]
Sep 18 19:14:41.555807 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.554957    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.557244 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.557169    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.559383 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.559360    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:14:41.561836 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.561815    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:41.561935 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.561881    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:14:41.564473 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.564003    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.566332 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.566312    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.568583 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.568565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.570980 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.570962    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.573305 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.573286    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:41.573391 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.573363    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:14:41.573492 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.573467    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:14:41.573553 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.573509    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:14:41.573758 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.573742    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:14:41.573832 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.573760    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:14:41.573832 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.573814    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:14:41.573930 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.573764    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:14:41.573981 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.573942    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:14:41.573981 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.573966    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:14:41.574915 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.574887    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:14:41.575003 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.574922    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:14:41.575141 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.575122    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:14:41.575192 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.575146    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:41.575192 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.575159    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:14:41.575454 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.575438    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:14:41.575543 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.575491    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:14:41.575763 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.575612    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:41.575763 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.575642    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:14:41.575763 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.575656    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jthc9\""
Sep 18 19:14:41.575763 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.575735    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:14:41.575962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.575880    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.576323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.576005    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s25tn\""
Sep 18 19:14:41.576323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.576036    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zvx4d\""
Sep 18 19:14:41.576323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.576040    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:14:41.576323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.576069    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:14:41.576323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.576175    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:14:41.576323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.576225    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:14:41.576323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.576226    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hrvvn\""
Sep 18 19:14:41.576684 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.576359    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ts62c\""
Sep 18 19:14:41.576684 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.576425    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5vbxn\""
Sep 18 19:14:41.576684 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.576451    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k7p47\""
Sep 18 19:14:41.578559 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.578536    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:41.581361 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.581341    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:41.581528 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.581497    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:14:41.582422 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.582406    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:41.584050 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.584033    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:14:41.587680 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.587656    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:41.587806 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.587784    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:14:41.588261 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.588239    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:14:41.588344 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.588291    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:14:41.588436 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.588414    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:41.588470 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.588447    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cdw6b\""
Sep 18 19:14:41.588677 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.588658    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8cjrm\""
Sep 18 19:14:41.593339 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593323    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-run-ovn\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.593480 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593425    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1d92004-612a-4adc-8159-2398bc926270-env-overrides\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.593480 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tlmp\" (UniqueName: \"kubernetes.io/projected/af4fccc9-2cf5-42f5-aa77-d7bc375a4c49-kube-api-access-9tlmp\") pod \"node-resolver-67hqf\" (UID: \"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49\") " pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.593480 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593477    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6132c829-7413-4429-99b7-6432b4e3c9da-host-slash\") pod \"iptables-alerter-hwb58\" (UID: \"6132c829-7413-4429-99b7-6432b4e3c9da\") " pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:41.593649 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593499    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-var-lib-cni-bin\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.593649 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42c37c95-3249-4102-a12a-8d903597aee0-multus-daemon-config\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.593649 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-sysctl-conf\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.593649 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5cb6368-a50c-4ec2-b30e-30effe2cf405-system-cni-dir\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.593649 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593613    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5cb6368-a50c-4ec2-b30e-30effe2cf405-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.593949 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-sys-fs\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.593949 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1d92004-612a-4adc-8159-2398bc926270-ovnkube-script-lib\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.593949 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af4fccc9-2cf5-42f5-aa77-d7bc375a4c49-hosts-file\") pod \"node-resolver-67hqf\" (UID: \"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49\") " pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.593949 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.593949 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-etc-openvswitch\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.593949 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-host\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.593949 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27xxb\" (UniqueName: \"kubernetes.io/projected/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-kube-api-access-27xxb\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.593945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-run-k8s-cni-cncf-io\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594036    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-run-netns\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-var-lib-kubelet\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-modprobe-d\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594111    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-lib-modules\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594154    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594177    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-cni-bin\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594190    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76d77ab0-46a5-432c-a812-67a2e57e0f07-serviceca\") pod \"node-ca-xfkrz\" (UID: \"76d77ab0-46a5-432c-a812-67a2e57e0f07\") " pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5cb6368-a50c-4ec2-b30e-30effe2cf405-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-run-netns\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594296    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-log-socket\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.594334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-cni-netd\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rv2\" (UniqueName: \"kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2\") pod \"network-check-target-5hs2m\" (UID: \"bf8dd3b4-3f67-4029-b116-ee69dcc738c6\") " pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-sysconfig\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594377    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-run\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-var-lib-kubelet\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-var-lib-openvswitch\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-etc-kubernetes\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76d77ab0-46a5-432c-a812-67a2e57e0f07-host\") pod \"node-ca-xfkrz\" (UID: \"76d77ab0-46a5-432c-a812-67a2e57e0f07\") " pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-kubernetes\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjtx\" (UniqueName: \"kubernetes.io/projected/cb350aae-48d3-4913-aec6-cd5f71e51350-kube-api-access-zrjtx\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594515    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-systemd-units\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1d92004-612a-4adc-8159-2398bc926270-ovn-node-metrics-cert\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594568    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42c37c95-3249-4102-a12a-8d903597aee0-cni-binary-copy\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-var-lib-cni-multus\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594638    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2gp\" (UniqueName: \"kubernetes.io/projected/6132c829-7413-4429-99b7-6432b4e3c9da-kube-api-access-4f2gp\") pod \"iptables-alerter-hwb58\" (UID: \"6132c829-7413-4429-99b7-6432b4e3c9da\") " pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594685    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-socket-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.594880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594724    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-node-log\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594764    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-run-ovn-kubernetes\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594803    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4fccc9-2cf5-42f5-aa77-d7bc375a4c49-tmp-dir\") pod \"node-resolver-67hqf\" (UID: \"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49\") " pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edf078c1-1bad-4f23-8753-bd4414735c17-dbus\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-tuned\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594889    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-etc-selinux\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6ldc\" (UniqueName: \"kubernetes.io/projected/a1d92004-612a-4adc-8159-2398bc926270-kube-api-access-j6ldc\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.594986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edf078c1-1bad-4f23-8753-bd4414735c17-kubelet-config\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-sysctl-d\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-systemd\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595085    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-multus-cni-dir\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-multus-socket-dir-parent\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595161    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blsm\" (UniqueName: \"kubernetes.io/projected/76d77ab0-46a5-432c-a812-67a2e57e0f07-kube-api-access-2blsm\") pod \"node-ca-xfkrz\" (UID: \"76d77ab0-46a5-432c-a812-67a2e57e0f07\") " pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5cb6368-a50c-4ec2-b30e-30effe2cf405-cnibin\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5cb6368-a50c-4ec2-b30e-30effe2cf405-cni-binary-copy\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.595647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595304    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-registration-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595328    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jql9v\" (UniqueName: \"kubernetes.io/projected/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-kube-api-access-jql9v\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595344    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-kubelet\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-run-systemd\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595371    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-system-cni-dir\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-sys\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595407    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1d92004-612a-4adc-8159-2398bc926270-ovnkube-config\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-cnibin\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-multus-conf-dir\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-run-multus-certs\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-tmp\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgkfv\" (UniqueName: \"kubernetes.io/projected/d5cb6368-a50c-4ec2-b30e-30effe2cf405-kube-api-access-rgkfv\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-device-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-run-openvswitch\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-hostroot\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwfv\" (UniqueName: \"kubernetes.io/projected/42c37c95-3249-4102-a12a-8d903597aee0-kube-api-access-5lwfv\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.596417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:41.597132 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5cb6368-a50c-4ec2-b30e-30effe2cf405-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.597132 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6132c829-7413-4429-99b7-6432b4e3c9da-iptables-alerter-script\") pod \"iptables-alerter-hwb58\" (UID: \"6132c829-7413-4429-99b7-6432b4e3c9da\") " pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:41.597132 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/273ddf75-7335-47f7-9bc7-c0c3f4727ebd-agent-certs\") pod \"konnectivity-agent-6lhsw\" (UID: \"273ddf75-7335-47f7-9bc7-c0c3f4727ebd\") " pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:14:41.597132 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/273ddf75-7335-47f7-9bc7-c0c3f4727ebd-konnectivity-ca\") pod \"konnectivity-agent-6lhsw\" (UID: \"273ddf75-7335-47f7-9bc7-c0c3f4727ebd\") " pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:14:41.597132 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.595820    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-slash\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.606392 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.606368    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:41.634396 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.634375    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-grhfz"
Sep 18 19:14:41.646428 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.646407    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-grhfz"
Sep 18 19:14:41.697406 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5cb6368-a50c-4ec2-b30e-30effe2cf405-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.697406 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697409    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-run-netns\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697605 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-log-socket\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697605 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-cni-netd\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697605 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rv2\" (UniqueName: \"kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2\") pod \"network-check-target-5hs2m\" (UID: \"bf8dd3b4-3f67-4029-b116-ee69dcc738c6\") " pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:41.697605 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697513    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-sysconfig\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.697605 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-run-netns\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697605 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697536    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-run\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.697605 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-var-lib-kubelet\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.697605 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697586    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-run\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697631    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-log-socket\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-var-lib-kubelet\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-cni-netd\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697690    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-sysconfig\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-var-lib-openvswitch\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-etc-kubernetes\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76d77ab0-46a5-432c-a812-67a2e57e0f07-host\") pod \"node-ca-xfkrz\" (UID: \"76d77ab0-46a5-432c-a812-67a2e57e0f07\") " pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-kubernetes\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-var-lib-openvswitch\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjtx\" (UniqueName: \"kubernetes.io/projected/cb350aae-48d3-4913-aec6-cd5f71e51350-kube-api-access-zrjtx\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76d77ab0-46a5-432c-a812-67a2e57e0f07-host\") pod \"node-ca-xfkrz\" (UID: \"76d77ab0-46a5-432c-a812-67a2e57e0f07\") " pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697847    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-systemd-units\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697879    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1d92004-612a-4adc-8159-2398bc926270-ovn-node-metrics-cert\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697903    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42c37c95-3249-4102-a12a-8d903597aee0-cni-binary-copy\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-var-lib-cni-multus\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697881    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-systemd-units\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.697962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.697970    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2gp\" (UniqueName: \"kubernetes.io/projected/6132c829-7413-4429-99b7-6432b4e3c9da-kube-api-access-4f2gp\") pod \"iptables-alerter-hwb58\" (UID: \"6132c829-7413-4429-99b7-6432b4e3c9da\") " pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-socket-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698067    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-etc-kubernetes\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-node-log\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698103    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-node-log\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-run-ovn-kubernetes\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698116    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-kubernetes\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4fccc9-2cf5-42f5-aa77-d7bc375a4c49-tmp-dir\") pod \"node-resolver-67hqf\" (UID: \"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49\") " pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edf078c1-1bad-4f23-8753-bd4414735c17-dbus\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698182    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-tuned\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-etc-selinux\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698239    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-socket-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698254    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5cb6368-a50c-4ec2-b30e-30effe2cf405-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edf078c1-1bad-4f23-8753-bd4414735c17-dbus\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6ldc\" (UniqueName: \"kubernetes.io/projected/a1d92004-612a-4adc-8159-2398bc926270-kube-api-access-j6ldc\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698335    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.698815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edf078c1-1bad-4f23-8753-bd4414735c17-kubelet-config\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698403    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-run-ovn-kubernetes\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-var-lib-cni-multus\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-etc-selinux\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edf078c1-1bad-4f23-8753-bd4414735c17-kubelet-config\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-sysctl-d\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-systemd\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698542    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-multus-cni-dir\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698565    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4fccc9-2cf5-42f5-aa77-d7bc375a4c49-tmp-dir\") pod \"node-resolver-67hqf\" (UID: \"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49\") " pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698581    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698594    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-multus-socket-dir-parent\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-multus-socket-dir-parent\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698620    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-systemd\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blsm\" (UniqueName: \"kubernetes.io/projected/76d77ab0-46a5-432c-a812-67a2e57e0f07-kube-api-access-2blsm\") pod \"node-ca-xfkrz\" (UID: \"76d77ab0-46a5-432c-a812-67a2e57e0f07\") " pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-multus-cni-dir\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5cb6368-a50c-4ec2-b30e-30effe2cf405-cnibin\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-sysctl-d\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698729    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5cb6368-a50c-4ec2-b30e-30effe2cf405-cnibin\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.699653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42c37c95-3249-4102-a12a-8d903597aee0-cni-binary-copy\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698837    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5cb6368-a50c-4ec2-b30e-30effe2cf405-cni-binary-copy\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-registration-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jql9v\" (UniqueName: \"kubernetes.io/projected/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-kube-api-access-jql9v\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.698973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-kubelet\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-run-systemd\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-system-cni-dir\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699063    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-sys\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1d92004-612a-4adc-8159-2398bc926270-ovnkube-config\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-cnibin\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-multus-conf-dir\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-run-multus-certs\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-tmp\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699292    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgkfv\" (UniqueName: \"kubernetes.io/projected/d5cb6368-a50c-4ec2-b30e-30effe2cf405-kube-api-access-rgkfv\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699308    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-registration-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699318    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-device-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699351    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-run-openvswitch\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.700327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699349    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5cb6368-a50c-4ec2-b30e-30effe2cf405-cni-binary-copy\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-hostroot\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwfv\" (UniqueName: \"kubernetes.io/projected/42c37c95-3249-4102-a12a-8d903597aee0-kube-api-access-5lwfv\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-run-systemd\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-device-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-kubelet\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-system-cni-dir\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699520    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5cb6368-a50c-4ec2-b30e-30effe2cf405-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699549    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6132c829-7413-4429-99b7-6432b4e3c9da-iptables-alerter-script\") pod \"iptables-alerter-hwb58\" (UID: \"6132c829-7413-4429-99b7-6432b4e3c9da\") " pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/273ddf75-7335-47f7-9bc7-c0c3f4727ebd-agent-certs\") pod \"konnectivity-agent-6lhsw\" (UID: \"273ddf75-7335-47f7-9bc7-c0c3f4727ebd\") " pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/273ddf75-7335-47f7-9bc7-c0c3f4727ebd-konnectivity-ca\") pod \"konnectivity-agent-6lhsw\" (UID: \"273ddf75-7335-47f7-9bc7-c0c3f4727ebd\") " pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-hostroot\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699651    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-run-multus-certs\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699672    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-slash\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699706    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-cnibin\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699746    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-multus-conf-dir\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5cb6368-a50c-4ec2-b30e-30effe2cf405-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.701071 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.699938    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1d92004-612a-4adc-8159-2398bc926270-ovnkube-config\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.700041    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret podName:edf078c1-1bad-4f23-8753-bd4414735c17 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:42.200019448 +0000 UTC m=+4.654157625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret") pod "global-pull-secret-syncer-pxfxm" (UID: "edf078c1-1bad-4f23-8753-bd4414735c17") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6132c829-7413-4429-99b7-6432b4e3c9da-iptables-alerter-script\") pod \"iptables-alerter-hwb58\" (UID: \"6132c829-7413-4429-99b7-6432b4e3c9da\") " pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.699624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-slash\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-run-ovn\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1d92004-612a-4adc-8159-2398bc926270-env-overrides\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700299    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-sys\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tlmp\" (UniqueName: \"kubernetes.io/projected/af4fccc9-2cf5-42f5-aa77-d7bc375a4c49-kube-api-access-9tlmp\") pod \"node-resolver-67hqf\" (UID: \"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49\") " pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700351    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-run-ovn\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700348    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6132c829-7413-4429-99b7-6432b4e3c9da-host-slash\") pod \"iptables-alerter-hwb58\" (UID: \"6132c829-7413-4429-99b7-6432b4e3c9da\") " pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-var-lib-cni-bin\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700392    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-run-openvswitch\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42c37c95-3249-4102-a12a-8d903597aee0-multus-daemon-config\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-sysctl-conf\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5cb6368-a50c-4ec2-b30e-30effe2cf405-system-cni-dir\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.701965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700484    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5cb6368-a50c-4ec2-b30e-30effe2cf405-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-sys-fs\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1d92004-612a-4adc-8159-2398bc926270-ovnkube-script-lib\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700565    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af4fccc9-2cf5-42f5-aa77-d7bc375a4c49-hosts-file\") pod \"node-resolver-67hqf\" (UID: \"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49\") " pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-etc-openvswitch\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-host\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700672    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xxb\" (UniqueName: \"kubernetes.io/projected/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-kube-api-access-27xxb\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-run-k8s-cni-cncf-io\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-run-netns\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-var-lib-kubelet\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-modprobe-d\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-lib-modules\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1d92004-612a-4adc-8159-2398bc926270-env-overrides\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.700994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-cni-bin\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76d77ab0-46a5-432c-a812-67a2e57e0f07-serviceca\") pod \"node-ca-xfkrz\" (UID: \"76d77ab0-46a5-432c-a812-67a2e57e0f07\") " pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/273ddf75-7335-47f7-9bc7-c0c3f4727ebd-konnectivity-ca\") pod \"konnectivity-agent-6lhsw\" (UID: \"273ddf75-7335-47f7-9bc7-c0c3f4727ebd\") " pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:14:41.702733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76d77ab0-46a5-432c-a812-67a2e57e0f07-serviceca\") pod \"node-ca-xfkrz\" (UID: \"76d77ab0-46a5-432c-a812-67a2e57e0f07\") " pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-etc-openvswitch\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-run-k8s-cni-cncf-io\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701666    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6132c829-7413-4429-99b7-6432b4e3c9da-host-slash\") pod \"iptables-alerter-hwb58\" (UID: \"6132c829-7413-4429-99b7-6432b4e3c9da\") " pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-var-lib-cni-bin\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701769    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-sys-fs\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701838    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-run-netns\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701886    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42c37c95-3249-4102-a12a-8d903597aee0-host-var-lib-kubelet\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.701982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-modprobe-d\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.702098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-lib-modules\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.702130    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5cb6368-a50c-4ec2-b30e-30effe2cf405-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.702186    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.702273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1d92004-612a-4adc-8159-2398bc926270-ovnkube-script-lib\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.702293    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs podName:cb350aae-48d3-4913-aec6-cd5f71e51350 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:42.202275463 +0000 UTC m=+4.656413620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs") pod "network-metrics-daemon-g9cck" (UID: "cb350aae-48d3-4913-aec6-cd5f71e51350") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.702340    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1d92004-612a-4adc-8159-2398bc926270-host-cni-bin\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.702395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af4fccc9-2cf5-42f5-aa77-d7bc375a4c49-hosts-file\") pod \"node-resolver-67hqf\" (UID: \"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49\") " pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.703417 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.702442    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-host\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.704182 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.702466    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-sysctl-conf\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.704182 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.702522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5cb6368-a50c-4ec2-b30e-30effe2cf405-system-cni-dir\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.704182 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.702784    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1d92004-612a-4adc-8159-2398bc926270-ovn-node-metrics-cert\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.704182 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.703251    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42c37c95-3249-4102-a12a-8d903597aee0-multus-daemon-config\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.704801 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.704772    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-tmp\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.704895 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.704818    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-etc-tuned\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.706566 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.706523    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/273ddf75-7335-47f7-9bc7-c0c3f4727ebd-agent-certs\") pod \"konnectivity-agent-6lhsw\" (UID: \"273ddf75-7335-47f7-9bc7-c0c3f4727ebd\") " pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:14:41.709447 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.709404    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal" event={"ID":"057a0ec2fb9c26d68b86efbdbfc5dfe9","Type":"ContainerStarted","Data":"aaa46619f1472a4c2f5b125127e869bf873561ff7d63c1ee2c5105ff92feb219"}
Sep 18 19:14:41.710944 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.710839    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal" event={"ID":"1464c738f5162da5f46c52a035e4932a","Type":"ContainerStarted","Data":"1e5f6430af43ab224eb8c526f422adee842d4b4ccf7f83ac81e4f376f9dcd41d"}
Sep 18 19:14:41.736675 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.736651    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:41.736675 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.736676    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:41.736872 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.736691    2559 projected.go:196] Error preparing data for projected volume kube-api-access-45rv2 for pod openshift-network-diagnostics/network-check-target-5hs2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:41.736872 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:41.736852    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2 podName:bf8dd3b4-3f67-4029-b116-ee69dcc738c6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:42.236815397 +0000 UTC m=+4.690953552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-45rv2" (UniqueName: "kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2") pod "network-check-target-5hs2m" (UID: "bf8dd3b4-3f67-4029-b116-ee69dcc738c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:41.739368 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.739343    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xxb\" (UniqueName: \"kubernetes.io/projected/d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5-kube-api-access-27xxb\") pod \"tuned-72js9\" (UID: \"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.739848 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.739825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgkfv\" (UniqueName: \"kubernetes.io/projected/d5cb6368-a50c-4ec2-b30e-30effe2cf405-kube-api-access-rgkfv\") pod \"multus-additional-cni-plugins-lhfb5\" (UID: \"d5cb6368-a50c-4ec2-b30e-30effe2cf405\") " pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.740008 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.739990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blsm\" (UniqueName: \"kubernetes.io/projected/76d77ab0-46a5-432c-a812-67a2e57e0f07-kube-api-access-2blsm\") pod \"node-ca-xfkrz\" (UID: \"76d77ab0-46a5-432c-a812-67a2e57e0f07\") " pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.740929 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.740836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jql9v\" (UniqueName: \"kubernetes.io/projected/332fc1df-98f6-4b34-b32a-d9dfd93d3ac4-kube-api-access-jql9v\") pod \"aws-ebs-csi-driver-node-dg26t\" (UID: \"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.741027 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.740990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjtx\" (UniqueName: \"kubernetes.io/projected/cb350aae-48d3-4913-aec6-cd5f71e51350-kube-api-access-zrjtx\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:41.741323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.741301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2gp\" (UniqueName: \"kubernetes.io/projected/6132c829-7413-4429-99b7-6432b4e3c9da-kube-api-access-4f2gp\") pod \"iptables-alerter-hwb58\" (UID: \"6132c829-7413-4429-99b7-6432b4e3c9da\") " pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:41.741416 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.741356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwfv\" (UniqueName: \"kubernetes.io/projected/42c37c95-3249-4102-a12a-8d903597aee0-kube-api-access-5lwfv\") pod \"multus-87xpd\" (UID: \"42c37c95-3249-4102-a12a-8d903597aee0\") " pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.742405 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.742389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tlmp\" (UniqueName: \"kubernetes.io/projected/af4fccc9-2cf5-42f5-aa77-d7bc375a4c49-kube-api-access-9tlmp\") pod \"node-resolver-67hqf\" (UID: \"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49\") " pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.748455 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.748437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6ldc\" (UniqueName: \"kubernetes.io/projected/a1d92004-612a-4adc-8159-2398bc926270-kube-api-access-j6ldc\") pod \"ovnkube-node-7hfmt\" (UID: \"a1d92004-612a-4adc-8159-2398bc926270\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.868535 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.868450    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xfkrz"
Sep 18 19:14:41.875861 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.875841    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t"
Sep 18 19:14:41.886468 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.886443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:14:41.894956 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.894934    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:14:41.902658 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.902637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-87xpd"
Sep 18 19:14:41.909283 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.909262    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-67hqf"
Sep 18 19:14:41.918049 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.918029    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-72js9"
Sep 18 19:14:41.926456 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.926432    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lhfb5"
Sep 18 19:14:41.932657 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:41.932638    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hwb58"
Sep 18 19:14:42.204589 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.204502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:42.204589 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.204570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:42.204808 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:42.204674    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:42.204808 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:42.204704    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:42.204808 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:42.204747    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret podName:edf078c1-1bad-4f23-8753-bd4414735c17 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:43.204727198 +0000 UTC m=+5.658865373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret") pod "global-pull-secret-syncer-pxfxm" (UID: "edf078c1-1bad-4f23-8753-bd4414735c17") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:42.204808 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:42.204763    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs podName:cb350aae-48d3-4913-aec6-cd5f71e51350 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:43.204756943 +0000 UTC m=+5.658895103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs") pod "network-metrics-daemon-g9cck" (UID: "cb350aae-48d3-4913-aec6-cd5f71e51350") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:42.305646 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.305613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rv2\" (UniqueName: \"kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2\") pod \"network-check-target-5hs2m\" (UID: \"bf8dd3b4-3f67-4029-b116-ee69dcc738c6\") " pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:42.305814 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:42.305768    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:42.305814 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:42.305788    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:42.305814 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:42.305798    2559 projected.go:196] Error preparing data for projected volume kube-api-access-45rv2 for pod openshift-network-diagnostics/network-check-target-5hs2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:42.305924 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:42.305848    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2 podName:bf8dd3b4-3f67-4029-b116-ee69dcc738c6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:43.305835896 +0000 UTC m=+5.759974072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-45rv2" (UniqueName: "kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2") pod "network-check-target-5hs2m" (UID: "bf8dd3b4-3f67-4029-b116-ee69dcc738c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:42.643439 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:42.643239    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4fccc9_2cf5_42f5_aa77_d7bc375a4c49.slice/crio-e9657c66cec9aa9fd1903ecc8dca28fc4ee6a75b7259c7b2a324eb8d3cd099e8 WatchSource:0}: Error finding container e9657c66cec9aa9fd1903ecc8dca28fc4ee6a75b7259c7b2a324eb8d3cd099e8: Status 404 returned error can't find the container with id e9657c66cec9aa9fd1903ecc8dca28fc4ee6a75b7259c7b2a324eb8d3cd099e8
Sep 18 19:14:42.644582 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:42.644560    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6132c829_7413_4429_99b7_6432b4e3c9da.slice/crio-33bd6dd42d34244e0c77aef6e455b65bf076945e3c66a8f0fea7ce603012cfcf WatchSource:0}: Error finding container 33bd6dd42d34244e0c77aef6e455b65bf076945e3c66a8f0fea7ce603012cfcf: Status 404 returned error can't find the container with id 33bd6dd42d34244e0c77aef6e455b65bf076945e3c66a8f0fea7ce603012cfcf
Sep 18 19:14:42.646195 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:42.646170    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273ddf75_7335_47f7_9bc7_c0c3f4727ebd.slice/crio-3c2cad34f8dc9130899bdc2db743db934464d3cc8cd8a624cfba8f1b106d905a WatchSource:0}: Error finding container 3c2cad34f8dc9130899bdc2db743db934464d3cc8cd8a624cfba8f1b106d905a: Status 404 returned error can't find the container with id 3c2cad34f8dc9130899bdc2db743db934464d3cc8cd8a624cfba8f1b106d905a
Sep 18 19:14:42.647661 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:42.647550    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d77ab0_46a5_432c_a812_67a2e57e0f07.slice/crio-1da559fb2905cb48a7c9889f71935f44da493087763a2484cd67a1f82cdcb721 WatchSource:0}: Error finding container 1da559fb2905cb48a7c9889f71935f44da493087763a2484cd67a1f82cdcb721: Status 404 returned error can't find the container with id 1da559fb2905cb48a7c9889f71935f44da493087763a2484cd67a1f82cdcb721
Sep 18 19:14:42.647895 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:42.647874    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c37c95_3249_4102_a12a_8d903597aee0.slice/crio-7e613481239558b7a7063d70f5558a1415dadd36ea34c8f33a7ae3da51e630e1 WatchSource:0}: Error finding container 7e613481239558b7a7063d70f5558a1415dadd36ea34c8f33a7ae3da51e630e1: Status 404 returned error can't find the container with id 7e613481239558b7a7063d70f5558a1415dadd36ea34c8f33a7ae3da51e630e1
Sep 18 19:14:42.648943 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.648864    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:41 +0000 UTC" deadline="2028-03-18 02:15:36.231288267 +0000 UTC"
Sep 18 19:14:42.648943 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.648893    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13111h0m53.582399553s"
Sep 18 19:14:42.650884 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:42.650704    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d92004_612a_4adc_8159_2398bc926270.slice/crio-db062ad41f374307f218a8d14b267ac98f1b498c565e566d8a61501e3224aad9 WatchSource:0}: Error finding container db062ad41f374307f218a8d14b267ac98f1b498c565e566d8a61501e3224aad9: Status 404 returned error can't find the container with id db062ad41f374307f218a8d14b267ac98f1b498c565e566d8a61501e3224aad9
Sep 18 19:14:42.672285 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:42.672254    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332fc1df_98f6_4b34_b32a_d9dfd93d3ac4.slice/crio-767aa6def616b80e2702e849368640362a8dc16371102ed9183d030f366ba69d WatchSource:0}: Error finding container 767aa6def616b80e2702e849368640362a8dc16371102ed9183d030f366ba69d: Status 404 returned error can't find the container with id 767aa6def616b80e2702e849368640362a8dc16371102ed9183d030f366ba69d
Sep 18 19:14:42.675820 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:14:42.675588    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c088ef_bcae_4e5f_a99d_0fb17f2d6fc5.slice/crio-8ca7ff7f5c3ea423838c9f671f8003ce60afda934570cdefd478fc06f116b035 WatchSource:0}: Error finding container 8ca7ff7f5c3ea423838c9f671f8003ce60afda934570cdefd478fc06f116b035: Status 404 returned error can't find the container with id 8ca7ff7f5c3ea423838c9f671f8003ce60afda934570cdefd478fc06f116b035
Sep 18 19:14:42.716087 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.716058    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6lhsw" event={"ID":"273ddf75-7335-47f7-9bc7-c0c3f4727ebd","Type":"ContainerStarted","Data":"3c2cad34f8dc9130899bdc2db743db934464d3cc8cd8a624cfba8f1b106d905a"}
Sep 18 19:14:42.717629 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.717610    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-72js9" event={"ID":"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5","Type":"ContainerStarted","Data":"8ca7ff7f5c3ea423838c9f671f8003ce60afda934570cdefd478fc06f116b035"}
Sep 18 19:14:42.718712 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.718685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hwb58" event={"ID":"6132c829-7413-4429-99b7-6432b4e3c9da","Type":"ContainerStarted","Data":"33bd6dd42d34244e0c77aef6e455b65bf076945e3c66a8f0fea7ce603012cfcf"}
Sep 18 19:14:42.719804 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.719784    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-67hqf" event={"ID":"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49","Type":"ContainerStarted","Data":"e9657c66cec9aa9fd1903ecc8dca28fc4ee6a75b7259c7b2a324eb8d3cd099e8"}
Sep 18 19:14:42.721017 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.720993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t" event={"ID":"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4","Type":"ContainerStarted","Data":"767aa6def616b80e2702e849368640362a8dc16371102ed9183d030f366ba69d"}
Sep 18 19:14:42.723222 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.723176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhfb5" event={"ID":"d5cb6368-a50c-4ec2-b30e-30effe2cf405","Type":"ContainerStarted","Data":"d560764a061f724443531c14f96a1a5a2ec1defaa5973224ff7cbbc072162ba1"}
Sep 18 19:14:42.726251 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.726225    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt" event={"ID":"a1d92004-612a-4adc-8159-2398bc926270","Type":"ContainerStarted","Data":"db062ad41f374307f218a8d14b267ac98f1b498c565e566d8a61501e3224aad9"}
Sep 18 19:14:42.727443 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.727419    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-87xpd" event={"ID":"42c37c95-3249-4102-a12a-8d903597aee0","Type":"ContainerStarted","Data":"7e613481239558b7a7063d70f5558a1415dadd36ea34c8f33a7ae3da51e630e1"}
Sep 18 19:14:42.728550 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:42.728531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xfkrz" event={"ID":"76d77ab0-46a5-432c-a812-67a2e57e0f07","Type":"ContainerStarted","Data":"1da559fb2905cb48a7c9889f71935f44da493087763a2484cd67a1f82cdcb721"}
Sep 18 19:14:43.211369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.211288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:43.211369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.211360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:43.211579 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.211514    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:43.211579 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.211576    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs podName:cb350aae-48d3-4913-aec6-cd5f71e51350 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:45.211556775 +0000 UTC m=+7.665694953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs") pod "network-metrics-daemon-g9cck" (UID: "cb350aae-48d3-4913-aec6-cd5f71e51350") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:43.212035 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.211944    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:43.212035 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.211991    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret podName:edf078c1-1bad-4f23-8753-bd4414735c17 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:45.211978534 +0000 UTC m=+7.666116692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret") pod "global-pull-secret-syncer-pxfxm" (UID: "edf078c1-1bad-4f23-8753-bd4414735c17") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:43.312239 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.312183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rv2\" (UniqueName: \"kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2\") pod \"network-check-target-5hs2m\" (UID: \"bf8dd3b4-3f67-4029-b116-ee69dcc738c6\") " pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:43.312408 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.312365    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:43.312408 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.312385    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:43.312408 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.312397    2559 projected.go:196] Error preparing data for projected volume kube-api-access-45rv2 for pod openshift-network-diagnostics/network-check-target-5hs2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:43.312563 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.312453    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2 podName:bf8dd3b4-3f67-4029-b116-ee69dcc738c6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:45.312434682 +0000 UTC m=+7.766572851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-45rv2" (UniqueName: "kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2") pod "network-check-target-5hs2m" (UID: "bf8dd3b4-3f67-4029-b116-ee69dcc738c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:43.649225 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.649146    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:41 +0000 UTC" deadline="2028-03-20 02:10:17.07209878 +0000 UTC"
Sep 18 19:14:43.649225 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.649218    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13158h55m33.422884596s"
Sep 18 19:14:43.707505 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.707469    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:43.707676 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.707610    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:14:43.707732 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.707695    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:43.708575 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.707778    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:14:43.708575 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.707845    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:43.708575 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:43.707924    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:14:43.738685 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.738644    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal" event={"ID":"057a0ec2fb9c26d68b86efbdbfc5dfe9","Type":"ContainerStarted","Data":"f553d8add9d5d9fbb71e5d6a4bd1fd753726a779c0ebf7ec1ed1596844f9b4b0"}
Sep 18 19:14:43.744050 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.744018    2559 generic.go:356] "Generic (PLEG): container finished" podID="1464c738f5162da5f46c52a035e4932a" containerID="e1716c55704655d430bebd9f2123a3e3467530030e571119d04a2312e10df897" exitCode=0
Sep 18 19:14:43.744180 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.744076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal" event={"ID":"1464c738f5162da5f46c52a035e4932a","Type":"ContainerDied","Data":"e1716c55704655d430bebd9f2123a3e3467530030e571119d04a2312e10df897"}
Sep 18 19:14:43.786596 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:43.786540    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-38.ec2.internal" podStartSLOduration=2.786520349 podStartE2EDuration="2.786520349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:43.762191299 +0000 UTC m=+6.216329487" watchObservedRunningTime="2026-09-18 19:14:43.786520349 +0000 UTC m=+6.240658528"
Sep 18 19:14:44.764762 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:44.764680    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal" event={"ID":"1464c738f5162da5f46c52a035e4932a","Type":"ContainerStarted","Data":"4e85e6b06ab9fd9b1875c07e9091cf4cc6f6de46747b24b6b127f1c1ab861be2"}
Sep 18 19:14:44.789304 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:44.789240    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-38.ec2.internal" podStartSLOduration=3.7892203159999998 podStartE2EDuration="3.789220316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:44.788429412 +0000 UTC m=+7.242567588" watchObservedRunningTime="2026-09-18 19:14:44.789220316 +0000 UTC m=+7.243358493"
Sep 18 19:14:45.229584 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:45.229313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:45.229584 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:45.229376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:45.229584 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.229519    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:45.229584 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.229595    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs podName:cb350aae-48d3-4913-aec6-cd5f71e51350 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:49.229561427 +0000 UTC m=+11.683699585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs") pod "network-metrics-daemon-g9cck" (UID: "cb350aae-48d3-4913-aec6-cd5f71e51350") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:45.230014 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.229996    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:45.230089 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.230052    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret podName:edf078c1-1bad-4f23-8753-bd4414735c17 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:49.230037355 +0000 UTC m=+11.684175513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret") pod "global-pull-secret-syncer-pxfxm" (UID: "edf078c1-1bad-4f23-8753-bd4414735c17") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:45.330568 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:45.330530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rv2\" (UniqueName: \"kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2\") pod \"network-check-target-5hs2m\" (UID: \"bf8dd3b4-3f67-4029-b116-ee69dcc738c6\") " pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:45.330756 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.330736    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:45.330756 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.330756    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:45.330861 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.330769    2559 projected.go:196] Error preparing data for projected volume kube-api-access-45rv2 for pod openshift-network-diagnostics/network-check-target-5hs2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:45.330861 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.330827    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2 podName:bf8dd3b4-3f67-4029-b116-ee69dcc738c6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:49.330807972 +0000 UTC m=+11.784946130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-45rv2" (UniqueName: "kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2") pod "network-check-target-5hs2m" (UID: "bf8dd3b4-3f67-4029-b116-ee69dcc738c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:45.706764 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:45.706296    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:45.706764 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.706424    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:14:45.706764 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:45.706477    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:45.706764 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.706564    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:14:45.706764 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:45.706600    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:45.706764 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:45.706654    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:14:47.705694 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:47.705642    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:47.705694 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:47.705659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:47.705694 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:47.705667    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:47.706349 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:47.705768    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:14:47.706349 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:47.705866    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:14:47.706349 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:47.706026    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:14:49.264065 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:49.264033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:49.264548 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:49.264081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:49.264548 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.264188    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:49.264548 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.264262    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs podName:cb350aae-48d3-4913-aec6-cd5f71e51350 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:57.264244579 +0000 UTC m=+19.718382748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs") pod "network-metrics-daemon-g9cck" (UID: "cb350aae-48d3-4913-aec6-cd5f71e51350") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:49.264711 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.264641    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:49.264711 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.264690    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret podName:edf078c1-1bad-4f23-8753-bd4414735c17 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:57.264675919 +0000 UTC m=+19.718814077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret") pod "global-pull-secret-syncer-pxfxm" (UID: "edf078c1-1bad-4f23-8753-bd4414735c17") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:49.365532 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:49.365435    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rv2\" (UniqueName: \"kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2\") pod \"network-check-target-5hs2m\" (UID: \"bf8dd3b4-3f67-4029-b116-ee69dcc738c6\") " pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:49.365703 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.365576    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:49.365703 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.365601    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:49.365703 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.365613    2559 projected.go:196] Error preparing data for projected volume kube-api-access-45rv2 for pod openshift-network-diagnostics/network-check-target-5hs2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:49.365703 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.365676    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2 podName:bf8dd3b4-3f67-4029-b116-ee69dcc738c6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:57.365659675 +0000 UTC m=+19.819797851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-45rv2" (UniqueName: "kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2") pod "network-check-target-5hs2m" (UID: "bf8dd3b4-3f67-4029-b116-ee69dcc738c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:49.706143 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:49.706111    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:49.706318 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:49.706157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:49.706318 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:49.706296    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:49.706433 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.706381    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:14:49.706745 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.706717    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:14:49.706840 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:49.706823    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:14:51.705653 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:51.705616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:51.706105 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:51.705625    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:51.706105 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:51.705625    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:51.706105 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:51.705792    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:14:51.706105 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:51.705943    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:14:51.706105 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:51.706081    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:14:53.705220 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:53.705173    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:53.705622 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:53.705171    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:53.705622 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:53.705308    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:14:53.705622 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:53.705389    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:53.705622 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:53.705386    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:14:53.705622 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:53.705487    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:14:55.705036 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:55.704985    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:55.705508 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:55.704985    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:55.705508 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:55.705117    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:14:55.705508 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:55.705240    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:14:55.705508 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:55.704996    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:55.705508 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:55.705350    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:14:57.324134 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:57.324098    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:57.324562 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:57.324177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:57.324562 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.324268    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:57.324562 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.324340    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs podName:cb350aae-48d3-4913-aec6-cd5f71e51350 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:13.32431999 +0000 UTC m=+35.778458155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs") pod "network-metrics-daemon-g9cck" (UID: "cb350aae-48d3-4913-aec6-cd5f71e51350") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:57.324562 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.324276    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:57.324562 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.324425    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret podName:edf078c1-1bad-4f23-8753-bd4414735c17 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:13.324407612 +0000 UTC m=+35.778545771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret") pod "global-pull-secret-syncer-pxfxm" (UID: "edf078c1-1bad-4f23-8753-bd4414735c17") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:57.425540 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:57.425504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rv2\" (UniqueName: \"kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2\") pod \"network-check-target-5hs2m\" (UID: \"bf8dd3b4-3f67-4029-b116-ee69dcc738c6\") " pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:57.425708 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.425664    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:57.425708 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.425689    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:57.425708 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.425701    2559 projected.go:196] Error preparing data for projected volume kube-api-access-45rv2 for pod openshift-network-diagnostics/network-check-target-5hs2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:57.425842 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.425751    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2 podName:bf8dd3b4-3f67-4029-b116-ee69dcc738c6 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:13.425739639 +0000 UTC m=+35.879877793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-45rv2" (UniqueName: "kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2") pod "network-check-target-5hs2m" (UID: "bf8dd3b4-3f67-4029-b116-ee69dcc738c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:57.705532 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:57.705450    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:57.705679 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:57.705454    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:57.705679 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.705572    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:14:57.705679 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:57.705455    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:57.705679 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.705665    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:14:57.705877 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:57.705777    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:14:59.706026 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:59.705988    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:14:59.706485 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:59.706065    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:14:59.706485 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:14:59.706106    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:14:59.706485 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:59.706114    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:14:59.706485 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:59.706178    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:14:59.706485 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:14:59.706283    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:15:00.797752 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.797379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t" event={"ID":"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4","Type":"ContainerStarted","Data":"06b5391a7eaa68ac5e41878e92c8a77eb7e24f9423abe2438b888c9a328560e0"}
Sep 18 19:15:00.799244 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.799214    2559 generic.go:356] "Generic (PLEG): container finished" podID="d5cb6368-a50c-4ec2-b30e-30effe2cf405" containerID="85ec2824aeea16c40595daf54eb8eb0df2a7566ebff185c16c37e4fd14daea26" exitCode=0
Sep 18 19:15:00.799388 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.799312    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhfb5" event={"ID":"d5cb6368-a50c-4ec2-b30e-30effe2cf405","Type":"ContainerDied","Data":"85ec2824aeea16c40595daf54eb8eb0df2a7566ebff185c16c37e4fd14daea26"}
Sep 18 19:15:00.804522 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.804493    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt" event={"ID":"a1d92004-612a-4adc-8159-2398bc926270","Type":"ContainerStarted","Data":"c4a1efdd295e1fa4d3ea247764cdb59fe353402a6b1d881f5d1ccc6f2a12345f"}
Sep 18 19:15:00.804622 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.804573    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt" event={"ID":"a1d92004-612a-4adc-8159-2398bc926270","Type":"ContainerStarted","Data":"47e17f38bd626ae450b663de6874cfc428b7264db22722baa85e0afdbb25cad3"}
Sep 18 19:15:00.804679 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.804626    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt" event={"ID":"a1d92004-612a-4adc-8159-2398bc926270","Type":"ContainerStarted","Data":"753792ecf6543d424d07d723df98f64c1a1424d62fd2095cfa30179f5fc61b6a"}
Sep 18 19:15:00.804729 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.804675    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt" event={"ID":"a1d92004-612a-4adc-8159-2398bc926270","Type":"ContainerStarted","Data":"27186f630c04c1d4e332d71e31899b688a49b2c337bfd062241351e58c855999"}
Sep 18 19:15:00.804784 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.804726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt" event={"ID":"a1d92004-612a-4adc-8159-2398bc926270","Type":"ContainerStarted","Data":"9027ff30007e2ecfb4a556c1adc039a9c8b8288e81dbc0c6991538835124b8a3"}
Sep 18 19:15:00.804784 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.804774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt" event={"ID":"a1d92004-612a-4adc-8159-2398bc926270","Type":"ContainerStarted","Data":"75e7c5b010b23d3039fa3665805e5f8dbc6f7c268e48a887000c125a61d4e792"}
Sep 18 19:15:00.806310 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.806288    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-87xpd" event={"ID":"42c37c95-3249-4102-a12a-8d903597aee0","Type":"ContainerStarted","Data":"91ba4e46509c62ec125e3733115c485bd8ec0e6144fdd02430ae93eac03aced2"}
Sep 18 19:15:00.808306 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.808277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xfkrz" event={"ID":"76d77ab0-46a5-432c-a812-67a2e57e0f07","Type":"ContainerStarted","Data":"633b2fdcd208bafee860fe74b110d3d804ef010bc419ee38ed5c27388e58223d"}
Sep 18 19:15:00.809849 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.809797    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6lhsw" event={"ID":"273ddf75-7335-47f7-9bc7-c0c3f4727ebd","Type":"ContainerStarted","Data":"f2b544d12b4de05b1ff783efcdab873fea9171a9687d37c522b26480e3c4f223"}
Sep 18 19:15:00.811484 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.811461    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-72js9" event={"ID":"d4c088ef-bcae-4e5f-a99d-0fb17f2d6fc5","Type":"ContainerStarted","Data":"8c72e01a01ae4d8867c65db43fbbb35494245b8302e38e8fe6ccc502876216e8"}
Sep 18 19:15:00.813055 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.813034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-67hqf" event={"ID":"af4fccc9-2cf5-42f5-aa77-d7bc375a4c49","Type":"ContainerStarted","Data":"6d72fe5137d6da923baecac91544a5e6675fcf54a1ee80888648ebc702e2d608"}
Sep 18 19:15:00.848356 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.848318    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xfkrz" podStartSLOduration=4.451745495 podStartE2EDuration="21.848306403s" totalImagesPullingTime="17.396560908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:00.847800965 +0000 UTC m=+23.301939143" watchObservedRunningTime="2026-09-18 19:15:00.848306403 +0000 UTC m=+23.302444579"
Sep 18 19:15:00.865001 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.864962    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6lhsw" podStartSLOduration=11.99417719 podStartE2EDuration="21.864949537s" totalImagesPullingTime="9.870772347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:00.864861773 +0000 UTC m=+23.318999951" watchObservedRunningTime="2026-09-18 19:15:00.864949537 +0000 UTC m=+23.319087713"
Sep 18 19:15:00.885845 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.885809    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-87xpd" podStartSLOduration=4.497400717 podStartE2EDuration="21.885801555s" totalImagesPullingTime="17.388400838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:00.885762266 +0000 UTC m=+23.339900443" watchObservedRunningTime="2026-09-18 19:15:00.885801555 +0000 UTC m=+23.339939761"
Sep 18 19:15:00.907612 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.907577    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-67hqf" podStartSLOduration=3.505224732 podStartE2EDuration="20.907568744s" totalImagesPullingTime="17.402344012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:00.907440868 +0000 UTC m=+23.361579058" watchObservedRunningTime="2026-09-18 19:15:00.907568744 +0000 UTC m=+23.361706921"
Sep 18 19:15:00.929110 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:00.929074    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-72js9" podStartSLOduration=3.559406457 podStartE2EDuration="20.92906273s" totalImagesPullingTime="17.369656273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:00.928557659 +0000 UTC m=+23.382695836" watchObservedRunningTime="2026-09-18 19:15:00.92906273 +0000 UTC m=+23.383200907"
Sep 18 19:15:01.222979 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:01.222960    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:15:01.647980 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:01.647862    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:15:01.222975053Z","UUID":"88cece06-0e5f-419c-b8da-2747632848d0","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:15:01.649758 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:01.649735    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:15:01.649896 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:01.649770    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:15:01.705842 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:01.705811    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:01.706017 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:01.705937    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:15:01.706017 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:01.705817    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:01.706136 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:01.706058    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:15:01.706136 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:01.706085    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:01.706260 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:01.706160    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:15:01.818144 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:01.818064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t" event={"ID":"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4","Type":"ContainerStarted","Data":"b37bf872eb7cf0a6a93af1e5be5b5f93600528f95b9a40a93936f2a5141d0130"}
Sep 18 19:15:01.820146 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:01.820088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hwb58" event={"ID":"6132c829-7413-4429-99b7-6432b4e3c9da","Type":"ContainerStarted","Data":"ef5ae66cf9aaa6423c890d69d4f71dafcf216bd69660254f4f6c752ac31be1cc"}
Sep 18 19:15:01.838758 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:01.838708    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hwb58" podStartSLOduration=4.437771877 podStartE2EDuration="21.838694952s" totalImagesPullingTime="17.400923075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:01.838447759 +0000 UTC m=+24.292585936" watchObservedRunningTime="2026-09-18 19:15:01.838694952 +0000 UTC m=+24.292833129"
Sep 18 19:15:02.825474 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:02.825436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t" event={"ID":"332fc1df-98f6-4b34-b32a-d9dfd93d3ac4","Type":"ContainerStarted","Data":"838595be68a0ab3455933c6ae1543c5b130bb3f469fc41871ab0a935bdcabcf8"}
Sep 18 19:15:02.830125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:02.830097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt" event={"ID":"a1d92004-612a-4adc-8159-2398bc926270","Type":"ContainerStarted","Data":"2e3658a35aaffbf8c32638ceceadaac869d0273190a8a8221398c00a2fbcabec"}
Sep 18 19:15:02.846995 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:02.846942    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dg26t" podStartSLOduration=4.515025817 podStartE2EDuration="23.846872034s" totalImagesPullingTime="19.331846217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:02.846374445 +0000 UTC m=+25.300512621" watchObservedRunningTime="2026-09-18 19:15:02.846872034 +0000 UTC m=+25.301010212"
Sep 18 19:15:03.705484 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:03.705453    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:03.705674 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:03.705553    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:15:03.705674 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:03.705592    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:03.705795 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:03.705703    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:15:03.705795 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:03.705754    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:03.705892 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:03.705834    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:15:05.676762 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.676509    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:15:05.677593 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.677278    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:15:05.705115 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.705087    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:05.705273 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:05.705171    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:15:05.705273 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.705185    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:05.705273 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.705273    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:05.705435 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:05.705384    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:15:05.705491 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:05.705472    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:15:05.839910 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.839878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt" event={"ID":"a1d92004-612a-4adc-8159-2398bc926270","Type":"ContainerStarted","Data":"57057bcfc0566c3a7f4109ef2815aeed006ba12078b00199236e21d063cc2a00"}
Sep 18 19:15:05.840084 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.840074    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:15:05.840151 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.840115    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:15:05.841884 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.841857    2559 generic.go:356] "Generic (PLEG): container finished" podID="d5cb6368-a50c-4ec2-b30e-30effe2cf405" containerID="25e03aac023ec814fef1d0449feb888289aee42275630140d32e796d7cf088f8" exitCode=0
Sep 18 19:15:05.841997 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.841931    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhfb5" event={"ID":"d5cb6368-a50c-4ec2-b30e-30effe2cf405","Type":"ContainerDied","Data":"25e03aac023ec814fef1d0449feb888289aee42275630140d32e796d7cf088f8"}
Sep 18 19:15:05.842097 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.842084    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:15:05.842961 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.842579    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6lhsw"
Sep 18 19:15:05.855689 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.855547    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:15:05.915084 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:05.915037    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt" podStartSLOduration=9.496677112 podStartE2EDuration="26.915022435s" totalImagesPullingTime="17.418345323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:05.886152301 +0000 UTC m=+28.340290478" watchObservedRunningTime="2026-09-18 19:15:05.915022435 +0000 UTC m=+28.369160611"
Sep 18 19:15:06.844765 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:06.844738    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:15:06.859726 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:06.859695    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:15:06.937602 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:06.937534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5hs2m"]
Sep 18 19:15:06.937733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:06.937640    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:06.937781 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:06.937731    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:15:06.941005 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:06.940980    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pxfxm"]
Sep 18 19:15:06.941130 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:06.941088    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:06.941185 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:06.941159    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:15:06.958316 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:06.958289    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g9cck"]
Sep 18 19:15:06.958437 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:06.958408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:06.958539 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:06.958519    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:15:07.848550 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:07.848520    2559 generic.go:356] "Generic (PLEG): container finished" podID="d5cb6368-a50c-4ec2-b30e-30effe2cf405" containerID="30a6caf79c09c12f8c16727a17ce33b51491d74ea6a0817e666a8b709d4e28d0" exitCode=0
Sep 18 19:15:07.848976 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:07.848604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhfb5" event={"ID":"d5cb6368-a50c-4ec2-b30e-30effe2cf405","Type":"ContainerDied","Data":"30a6caf79c09c12f8c16727a17ce33b51491d74ea6a0817e666a8b709d4e28d0"}
Sep 18 19:15:08.705285 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:08.705255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:08.705514 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:08.705255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:08.705514 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:08.705365    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:15:08.705514 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:08.705255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:08.705514 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:08.705455    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:15:08.705684 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:08.705551    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:15:09.854982 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:09.854721    2559 generic.go:356] "Generic (PLEG): container finished" podID="d5cb6368-a50c-4ec2-b30e-30effe2cf405" containerID="507f3d63bbde132c8a9e258c569d9962560f967911b9038cb035b39f52f8f740" exitCode=0
Sep 18 19:15:09.855372 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:09.854798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhfb5" event={"ID":"d5cb6368-a50c-4ec2-b30e-30effe2cf405","Type":"ContainerDied","Data":"507f3d63bbde132c8a9e258c569d9962560f967911b9038cb035b39f52f8f740"}
Sep 18 19:15:10.704910 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:10.704877    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:10.705102 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:10.704882    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:10.705102 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:10.705022    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:10.705102 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:10.704992    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:15:10.705299 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:10.705113    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:15:10.705299 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:10.705214    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:15:12.705107 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:12.705073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:12.705107 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:12.705106    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:12.705688 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:12.705112    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:12.705688 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:12.705193    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2m" podUID="bf8dd3b4-3f67-4029-b116-ee69dcc738c6"
Sep 18 19:15:12.705688 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:12.705259    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxfxm" podUID="edf078c1-1bad-4f23-8753-bd4414735c17"
Sep 18 19:15:12.705688 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:12.705338    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9cck" podUID="cb350aae-48d3-4913-aec6-cd5f71e51350"
Sep 18 19:15:12.997321 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:12.997244    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-38.ec2.internal" event="NodeReady"
Sep 18 19:15:12.997462 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:12.997381    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:15:13.053390 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.053358    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fqqh8"]
Sep 18 19:15:13.080735 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.080699    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vq6nl"]
Sep 18 19:15:13.080899 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.080833    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.084572 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.084547    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:15:13.084572 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.084566    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:15:13.084725 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.084657    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sd5f2\""
Sep 18 19:15:13.108252 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.108230    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fqqh8"]
Sep 18 19:15:13.108364 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.108269    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vq6nl"]
Sep 18 19:15:13.108418 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.108360    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vq6nl"
Sep 18 19:15:13.111737 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.111714    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:15:13.112059 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.112043    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:15:13.112493 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.112474    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:15:13.115645 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.115627    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lxlj8\""
Sep 18 19:15:13.242104 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.241938    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.242104 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.241986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe654a7-76ea-4741-ae34-d2b4566cf622-config-volume\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.242104 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.242039    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe654a7-76ea-4741-ae34-d2b4566cf622-tmp-dir\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.242104 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.242068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4af3fc5f-2070-42ee-a588-c4197598b28a-cert\") pod \"ingress-canary-vq6nl\" (UID: \"4af3fc5f-2070-42ee-a588-c4197598b28a\") " pod="openshift-ingress-canary/ingress-canary-vq6nl"
Sep 18 19:15:13.242104 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.242087    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbhh\" (UniqueName: \"kubernetes.io/projected/fbe654a7-76ea-4741-ae34-d2b4566cf622-kube-api-access-vvbhh\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.242494 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.242188    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hbb\" (UniqueName: \"kubernetes.io/projected/4af3fc5f-2070-42ee-a588-c4197598b28a-kube-api-access-84hbb\") pod \"ingress-canary-vq6nl\" (UID: \"4af3fc5f-2070-42ee-a588-c4197598b28a\") " pod="openshift-ingress-canary/ingress-canary-vq6nl"
Sep 18 19:15:13.343345 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.343311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe654a7-76ea-4741-ae34-d2b4566cf622-tmp-dir\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.343532 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.343355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4af3fc5f-2070-42ee-a588-c4197598b28a-cert\") pod \"ingress-canary-vq6nl\" (UID: \"4af3fc5f-2070-42ee-a588-c4197598b28a\") " pod="openshift-ingress-canary/ingress-canary-vq6nl"
Sep 18 19:15:13.343532 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.343384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbhh\" (UniqueName: \"kubernetes.io/projected/fbe654a7-76ea-4741-ae34-d2b4566cf622-kube-api-access-vvbhh\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.343721 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.343699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe654a7-76ea-4741-ae34-d2b4566cf622-tmp-dir\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.343778 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.343743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hbb\" (UniqueName: \"kubernetes.io/projected/4af3fc5f-2070-42ee-a588-c4197598b28a-kube-api-access-84hbb\") pod \"ingress-canary-vq6nl\" (UID: \"4af3fc5f-2070-42ee-a588-c4197598b28a\") " pod="openshift-ingress-canary/ingress-canary-vq6nl"
Sep 18 19:15:13.343825 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.343793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:13.343825 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.343816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.343903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.343849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe654a7-76ea-4741-ae34-d2b4566cf622-config-volume\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.343903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.343890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:13.344135 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.344017    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:13.344135 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.344017    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:15:13.344135 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.344086    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls podName:fbe654a7-76ea-4741-ae34-d2b4566cf622 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:13.844067335 +0000 UTC m=+36.298205511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls") pod "dns-default-fqqh8" (UID: "fbe654a7-76ea-4741-ae34-d2b4566cf622") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:13.344135 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.344128    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret podName:edf078c1-1bad-4f23-8753-bd4414735c17 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:45.344110904 +0000 UTC m=+67.798249079 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret") pod "global-pull-secret-syncer-pxfxm" (UID: "edf078c1-1bad-4f23-8753-bd4414735c17") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:15:13.344135 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.344074    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:15:13.344394 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.344168    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs podName:cb350aae-48d3-4913-aec6-cd5f71e51350 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:45.34415839 +0000 UTC m=+67.798296563 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs") pod "network-metrics-daemon-g9cck" (UID: "cb350aae-48d3-4913-aec6-cd5f71e51350") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:15:13.348378 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.348346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4af3fc5f-2070-42ee-a588-c4197598b28a-cert\") pod \"ingress-canary-vq6nl\" (UID: \"4af3fc5f-2070-42ee-a588-c4197598b28a\") " pod="openshift-ingress-canary/ingress-canary-vq6nl"
Sep 18 19:15:13.353077 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.353045    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbhh\" (UniqueName: \"kubernetes.io/projected/fbe654a7-76ea-4741-ae34-d2b4566cf622-kube-api-access-vvbhh\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.353191 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.353162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hbb\" (UniqueName: \"kubernetes.io/projected/4af3fc5f-2070-42ee-a588-c4197598b28a-kube-api-access-84hbb\") pod \"ingress-canary-vq6nl\" (UID: \"4af3fc5f-2070-42ee-a588-c4197598b28a\") " pod="openshift-ingress-canary/ingress-canary-vq6nl"
Sep 18 19:15:13.353774 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.353753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe654a7-76ea-4741-ae34-d2b4566cf622-config-volume\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.417685 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.417649    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vq6nl"
Sep 18 19:15:13.445255 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.445229    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rv2\" (UniqueName: \"kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2\") pod \"network-check-target-5hs2m\" (UID: \"bf8dd3b4-3f67-4029-b116-ee69dcc738c6\") " pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:13.445392 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.445370    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:15:13.445493 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.445397    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:15:13.445493 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.445410    2559 projected.go:196] Error preparing data for projected volume kube-api-access-45rv2 for pod openshift-network-diagnostics/network-check-target-5hs2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:15:13.445493 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.445465    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2 podName:bf8dd3b4-3f67-4029-b116-ee69dcc738c6 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:45.445445765 +0000 UTC m=+67.899583935 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-45rv2" (UniqueName: "kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2") pod "network-check-target-5hs2m" (UID: "bf8dd3b4-3f67-4029-b116-ee69dcc738c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:15:13.849453 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:13.849419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:13.850070 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.849587    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:13.850070 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:13.849646    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls podName:fbe654a7-76ea-4741-ae34-d2b4566cf622 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:14.849633127 +0000 UTC m=+37.303771281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls") pod "dns-default-fqqh8" (UID: "fbe654a7-76ea-4741-ae34-d2b4566cf622") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:14.405400 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.405171    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vq6nl"]
Sep 18 19:15:14.408678 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:14.408645    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af3fc5f_2070_42ee_a588_c4197598b28a.slice/crio-4f1c9253a055060a92610d42820cc363cbed4f1e82f573b7fc844663ac8053f3 WatchSource:0}: Error finding container 4f1c9253a055060a92610d42820cc363cbed4f1e82f573b7fc844663ac8053f3: Status 404 returned error can't find the container with id 4f1c9253a055060a92610d42820cc363cbed4f1e82f573b7fc844663ac8053f3
Sep 18 19:15:14.705443 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.705324    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:14.705443 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.705359    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:14.705443 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.705360    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:14.716728 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.716696    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:15:14.716728 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.716696    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:15:14.718668 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.718168    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:15:14.718668 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.718291    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:15:14.718668 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.718382    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-95w6x\""
Sep 18 19:15:14.718668 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.718550    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lf5lk\""
Sep 18 19:15:14.856854 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.856823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:14.857307 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:14.856966    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:14.857307 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:14.857030    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls podName:fbe654a7-76ea-4741-ae34-d2b4566cf622 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:16.857014428 +0000 UTC m=+39.311152584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls") pod "dns-default-fqqh8" (UID: "fbe654a7-76ea-4741-ae34-d2b4566cf622") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:14.870325 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.870300    2559 generic.go:356] "Generic (PLEG): container finished" podID="d5cb6368-a50c-4ec2-b30e-30effe2cf405" containerID="05b354afa250545deeb57d941e892f5dc2e403a2722fee2cbffed7835c218db7" exitCode=0
Sep 18 19:15:14.870440 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.870383    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhfb5" event={"ID":"d5cb6368-a50c-4ec2-b30e-30effe2cf405","Type":"ContainerDied","Data":"05b354afa250545deeb57d941e892f5dc2e403a2722fee2cbffed7835c218db7"}
Sep 18 19:15:14.871542 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:14.871522    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vq6nl" event={"ID":"4af3fc5f-2070-42ee-a588-c4197598b28a","Type":"ContainerStarted","Data":"4f1c9253a055060a92610d42820cc363cbed4f1e82f573b7fc844663ac8053f3"}
Sep 18 19:15:15.877762 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:15.877727    2559 generic.go:356] "Generic (PLEG): container finished" podID="d5cb6368-a50c-4ec2-b30e-30effe2cf405" containerID="bf5e62a7841868778b19c333be1f08f3332c993a44dfb7df9218cb5aea8777cb" exitCode=0
Sep 18 19:15:15.878177 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:15.877799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhfb5" event={"ID":"d5cb6368-a50c-4ec2-b30e-30effe2cf405","Type":"ContainerDied","Data":"bf5e62a7841868778b19c333be1f08f3332c993a44dfb7df9218cb5aea8777cb"}
Sep 18 19:15:16.874691 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:16.874664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:16.874834 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:16.874816    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:16.874890 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:16.874881    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls podName:fbe654a7-76ea-4741-ae34-d2b4566cf622 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:20.874863391 +0000 UTC m=+43.329001545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls") pod "dns-default-fqqh8" (UID: "fbe654a7-76ea-4741-ae34-d2b4566cf622") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:16.884727 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:16.884704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhfb5" event={"ID":"d5cb6368-a50c-4ec2-b30e-30effe2cf405","Type":"ContainerStarted","Data":"30478825c3eb5dee8442febcd0627f5765e5932139ca7b0cd8afd65d30093213"}
Sep 18 19:15:16.916221 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:16.916163    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lhfb5" podStartSLOduration=8.411933934 podStartE2EDuration="36.916147255s" totalImagesPullingTime="28.504213321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:16.915617952 +0000 UTC m=+39.369756128" watchObservedRunningTime="2026-09-18 19:15:16.916147255 +0000 UTC m=+39.370285433"
Sep 18 19:15:17.891810 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:17.891607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vq6nl" event={"ID":"4af3fc5f-2070-42ee-a588-c4197598b28a","Type":"ContainerStarted","Data":"1a5155fc7c414649a594f8d660f12ac814823b34982d33ae50867fab068ff0a1"}
Sep 18 19:15:17.911092 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:17.911046    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vq6nl" podStartSLOduration=2.478545265 podStartE2EDuration="4.911033153s" totalImagesPullingTime="2.432487888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:17.909687205 +0000 UTC m=+40.363825381" watchObservedRunningTime="2026-09-18 19:15:17.911033153 +0000 UTC m=+40.365171338"
Sep 18 19:15:20.902731 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:20.902694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:20.903116 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:20.902800    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:20.903116 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:20.902848    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls podName:fbe654a7-76ea-4741-ae34-d2b4566cf622 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:28.902836058 +0000 UTC m=+51.356974212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls") pod "dns-default-fqqh8" (UID: "fbe654a7-76ea-4741-ae34-d2b4566cf622") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:24.160102 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.160066    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-kfskl"]
Sep 18 19:15:24.161987 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.161972    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.164484 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.164446    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:15:24.164617 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.164479    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 19:15:24.164617 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.164490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 19:15:24.165513 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.165493    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cnlzg\""
Sep 18 19:15:24.165663 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.165493    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:15:24.171067 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.171051    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 19:15:24.173030 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.173006    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-kfskl"]
Sep 18 19:15:24.288047 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.288022    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-58ccb95cb-jb24t"]
Sep 18 19:15:24.289764 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.289750    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.294238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.294190    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 19:15:24.295410 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.295393    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 19:15:24.295652 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.295632    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 19:15:24.295778 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.295754    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8nw9x\""
Sep 18 19:15:24.295862 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.295639    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 19:15:24.295934 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.295906    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 19:15:24.298963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.298947    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 19:15:24.304793 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.304775    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58ccb95cb-jb24t"]
Sep 18 19:15:24.326407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.326381    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.326495 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.326441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.326495 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.326479    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsdpt\" (UniqueName: \"kubernetes.io/projected/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-kube-api-access-bsdpt\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.326570 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.326521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-tmp\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.326570 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.326544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-serving-cert\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.326570 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.326562    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-snapshots\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.404041 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.404021    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"]
Sep 18 19:15:24.405744 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.405731    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.408848 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.408830    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 19:15:24.408848 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.408839    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 19:15:24.408999 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.408866    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 19:15:24.408999 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.408839    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:15:24.414785 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.414768    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-v99j6\""
Sep 18 19:15:24.427769 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.427746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-snapshots\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.427859 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.427780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-default-certificate\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.427859 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.427817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.427859 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.427832    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.428005 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.427877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.428005 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.427892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-stats-auth\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.428005 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.427910    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.428005 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.427931    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzl59\" (UniqueName: \"kubernetes.io/projected/51a4f6b3-247d-4614-976d-e585479998d7-kube-api-access-gzl59\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.428005 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.427971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsdpt\" (UniqueName: \"kubernetes.io/projected/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-kube-api-access-bsdpt\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.428005 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.428001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-tmp\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.428284 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.428030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-serving-cert\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.428432 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.428412    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-tmp\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.428522 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.428502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-snapshots\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.429093 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.429072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.429255 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.429072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.431720 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.431702    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-serving-cert\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.433031 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.433011    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"]
Sep 18 19:15:24.451807 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.451789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsdpt\" (UniqueName: \"kubernetes.io/projected/b6899294-bd1e-4b9c-ab8a-f10dfaba7a94-kube-api-access-bsdpt\") pod \"insights-operator-6f9c644dd9-kfskl\" (UID: \"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94\") " pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.471604 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.471584    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-kfskl"
Sep 18 19:15:24.520161 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.519893    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-b9dcv"]
Sep 18 19:15:24.522011 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.521988    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.529715 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.528146    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 19:15:24.529715 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.528764    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-default-certificate\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.529715 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.528813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fbe7c860-1b27-480f-9de5-ba6eddfb76a0-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-92gfp\" (UID: \"fbe7c860-1b27-480f-9de5-ba6eddfb76a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.529715 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.528898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.529715 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.528936    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7vz\" (UniqueName: \"kubernetes.io/projected/fbe7c860-1b27-480f-9de5-ba6eddfb76a0-kube-api-access-lb7vz\") pod \"kube-storage-version-migrator-operator-55889f7b47-92gfp\" (UID: \"fbe7c860-1b27-480f-9de5-ba6eddfb76a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.529715 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.528973    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fbe7c860-1b27-480f-9de5-ba6eddfb76a0-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-92gfp\" (UID: \"fbe7c860-1b27-480f-9de5-ba6eddfb76a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.529715 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.529045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-stats-auth\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.529715 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.529081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.529715 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.529111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzl59\" (UniqueName: \"kubernetes.io/projected/51a4f6b3-247d-4614-976d-e585479998d7-kube-api-access-gzl59\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.530230 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.529749    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 19:15:24.530230 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.529814    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:15:24.530230 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:24.530106    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:25.030076822 +0000 UTC m=+47.484214990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:24.530418 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:24.530400    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:24.532290 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:24.531437    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:25.031415112 +0000 UTC m=+47.485553281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : secret "router-metrics-certs-default" not found
Sep 18 19:15:24.533860 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.533128    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 19:15:24.534900 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.534390    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-86tfz\""
Sep 18 19:15:24.539533 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.539490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-stats-auth\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.540057 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.540039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-default-certificate\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.541410 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.541272    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 19:15:24.543086 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.543054    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-b9dcv"]
Sep 18 19:15:24.548059 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.548043    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzl59\" (UniqueName: \"kubernetes.io/projected/51a4f6b3-247d-4614-976d-e585479998d7-kube-api-access-gzl59\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:24.629658 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.629632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-serving-cert\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.629815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.629674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fbe7c860-1b27-480f-9de5-ba6eddfb76a0-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-92gfp\" (UID: \"fbe7c860-1b27-480f-9de5-ba6eddfb76a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.629815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.629699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-trusted-ca\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.629815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.629754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7vz\" (UniqueName: \"kubernetes.io/projected/fbe7c860-1b27-480f-9de5-ba6eddfb76a0-kube-api-access-lb7vz\") pod \"kube-storage-version-migrator-operator-55889f7b47-92gfp\" (UID: \"fbe7c860-1b27-480f-9de5-ba6eddfb76a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.629815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.629780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fbe7c860-1b27-480f-9de5-ba6eddfb76a0-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-92gfp\" (UID: \"fbe7c860-1b27-480f-9de5-ba6eddfb76a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.630025 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.629922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-config\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.630025 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.629952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6q2x\" (UniqueName: \"kubernetes.io/projected/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-kube-api-access-t6q2x\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.630324 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.630307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fbe7c860-1b27-480f-9de5-ba6eddfb76a0-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-92gfp\" (UID: \"fbe7c860-1b27-480f-9de5-ba6eddfb76a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.631853 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.631830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fbe7c860-1b27-480f-9de5-ba6eddfb76a0-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-92gfp\" (UID: \"fbe7c860-1b27-480f-9de5-ba6eddfb76a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.641636 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.641613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7vz\" (UniqueName: \"kubernetes.io/projected/fbe7c860-1b27-480f-9de5-ba6eddfb76a0-kube-api-access-lb7vz\") pod \"kube-storage-version-migrator-operator-55889f7b47-92gfp\" (UID: \"fbe7c860-1b27-480f-9de5-ba6eddfb76a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.714684 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.714625    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"
Sep 18 19:15:24.731048 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.731023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-trusted-ca\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.731153 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.731080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-config\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.731153 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.731098    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6q2x\" (UniqueName: \"kubernetes.io/projected/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-kube-api-access-t6q2x\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.731153 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.731150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-serving-cert\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.731712 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.731690    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-config\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.731938 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.731917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-trusted-ca\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.733352 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.733331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-serving-cert\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.754415 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.754394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6q2x\" (UniqueName: \"kubernetes.io/projected/cc94edce-8cdd-40dc-bd9e-68cc15000b6a-kube-api-access-t6q2x\") pod \"console-operator-fd4bb46c-b9dcv\" (UID: \"cc94edce-8cdd-40dc-bd9e-68cc15000b6a\") " pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.760806 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.760753    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-kfskl"]
Sep 18 19:15:24.768712 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:24.768684    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6899294_bd1e_4b9c_ab8a_f10dfaba7a94.slice/crio-8ae37e8e6016ec5961dc03973b29551c1d4f0c9d0981c22c666028bd005a28cf WatchSource:0}: Error finding container 8ae37e8e6016ec5961dc03973b29551c1d4f0c9d0981c22c666028bd005a28cf: Status 404 returned error can't find the container with id 8ae37e8e6016ec5961dc03973b29551c1d4f0c9d0981c22c666028bd005a28cf
Sep 18 19:15:24.837851 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.837826    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:24.908114 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.908090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-kfskl" event={"ID":"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94","Type":"ContainerStarted","Data":"8ae37e8e6016ec5961dc03973b29551c1d4f0c9d0981c22c666028bd005a28cf"}
Sep 18 19:15:24.984257 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:24.984231    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp"]
Sep 18 19:15:24.986361 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:24.986338    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe7c860_1b27_480f_9de5_ba6eddfb76a0.slice/crio-b18467311349c5633e697ec63a13112c9a73ac70842f251c9033d66dd287d126 WatchSource:0}: Error finding container b18467311349c5633e697ec63a13112c9a73ac70842f251c9033d66dd287d126: Status 404 returned error can't find the container with id b18467311349c5633e697ec63a13112c9a73ac70842f251c9033d66dd287d126
Sep 18 19:15:25.032832 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:25.032807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:25.032916 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:25.032866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:25.032955 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:25.032922    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:26.032905822 +0000 UTC m=+48.487043990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:25.032996 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:25.032975    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:25.033031 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:25.033009    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:26.032999495 +0000 UTC m=+48.487137649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : secret "router-metrics-certs-default" not found
Sep 18 19:15:25.109259 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:25.109230    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-b9dcv"]
Sep 18 19:15:25.111733 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:25.111710    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc94edce_8cdd_40dc_bd9e_68cc15000b6a.slice/crio-b133edf3eb844a0a72a59a658847abf37194bf95ac19d3d6adae605eda37f4be WatchSource:0}: Error finding container b133edf3eb844a0a72a59a658847abf37194bf95ac19d3d6adae605eda37f4be: Status 404 returned error can't find the container with id b133edf3eb844a0a72a59a658847abf37194bf95ac19d3d6adae605eda37f4be
Sep 18 19:15:25.913522 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:25.913452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp" event={"ID":"fbe7c860-1b27-480f-9de5-ba6eddfb76a0","Type":"ContainerStarted","Data":"b18467311349c5633e697ec63a13112c9a73ac70842f251c9033d66dd287d126"}
Sep 18 19:15:25.915197 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:25.915138    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv" event={"ID":"cc94edce-8cdd-40dc-bd9e-68cc15000b6a","Type":"ContainerStarted","Data":"b133edf3eb844a0a72a59a658847abf37194bf95ac19d3d6adae605eda37f4be"}
Sep 18 19:15:26.042790 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:26.042750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:26.042978 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:26.042833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:26.042978 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:26.042944    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:26.043103 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:26.042995    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:28.042975813 +0000 UTC m=+50.497113975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:26.043103 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:26.043018    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:28.043008493 +0000 UTC m=+50.497146648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : secret "router-metrics-certs-default" not found
Sep 18 19:15:27.594465 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.594430    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"]
Sep 18 19:15:27.596587 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.596564    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:27.599248 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.599224    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 19:15:27.599530 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.599510    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tmph6\""
Sep 18 19:15:27.599602 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.599523    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 19:15:27.609127 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.609108    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"]
Sep 18 19:15:27.759564 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.759538    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:27.759675 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.759589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/048b4230-fb95-4853-8eaf-830198c66298-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:27.860363 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.860254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:27.860363 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.860317    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/048b4230-fb95-4853-8eaf-830198c66298-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:27.860567 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:27.860417    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:27.860567 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:27.860498    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert podName:048b4230-fb95-4853-8eaf-830198c66298 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:28.360478515 +0000 UTC m=+50.814616686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-bqhhx" (UID: "048b4230-fb95-4853-8eaf-830198c66298") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:27.861141 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.861109    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/048b4230-fb95-4853-8eaf-830198c66298-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:27.922166 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.922130    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp" event={"ID":"fbe7c860-1b27-480f-9de5-ba6eddfb76a0","Type":"ContainerStarted","Data":"c22aa5f8bbff5563e76e92f646b2ad1c9e9b3a529b8f450ed5a83e5a67332627"}
Sep 18 19:15:27.923795 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.923764    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-kfskl" event={"ID":"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94","Type":"ContainerStarted","Data":"f1da04ec05ba8db9a027d10aea7e4e0c7f4282b543a23295cc72d5f322a25ec7"}
Sep 18 19:15:27.925411 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.925392    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/0.log"
Sep 18 19:15:27.925518 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.925427    2559 generic.go:356] "Generic (PLEG): container finished" podID="cc94edce-8cdd-40dc-bd9e-68cc15000b6a" containerID="5ec228d7a5e9c0eca119e6eb4c75a548695d775fb682d796ab2f973b5260baa4" exitCode=255
Sep 18 19:15:27.925518 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.925469    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv" event={"ID":"cc94edce-8cdd-40dc-bd9e-68cc15000b6a","Type":"ContainerDied","Data":"5ec228d7a5e9c0eca119e6eb4c75a548695d775fb682d796ab2f973b5260baa4"}
Sep 18 19:15:27.925682 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.925665    2559 scope.go:117] "RemoveContainer" containerID="5ec228d7a5e9c0eca119e6eb4c75a548695d775fb682d796ab2f973b5260baa4"
Sep 18 19:15:27.941623 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:27.940664    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp" podStartSLOduration=1.175613117 podStartE2EDuration="3.940640267s" totalImagesPullingTime="2.76502715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:27.940502572 +0000 UTC m=+50.394640749" watchObservedRunningTime="2026-09-18 19:15:27.940640267 +0000 UTC m=+50.394778444"
Sep 18 19:15:28.062429 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.062393    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:28.062579 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:28.062562    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:28.062641 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:28.062626    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:32.062607956 +0000 UTC m=+54.516746136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : secret "router-metrics-certs-default" not found
Sep 18 19:15:28.063144 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.063120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:28.063290 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:28.063236    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:32.06321737 +0000 UTC m=+54.517355543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:28.365650 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.365613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:28.365843 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:28.365769    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:28.365843 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:28.365836    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert podName:048b4230-fb95-4853-8eaf-830198c66298 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:29.36582154 +0000 UTC m=+51.819959695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-bqhhx" (UID: "048b4230-fb95-4853-8eaf-830198c66298") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:28.929212 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.929175    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:15:28.929684 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.929647    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/0.log"
Sep 18 19:15:28.929755 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.929686    2559 generic.go:356] "Generic (PLEG): container finished" podID="cc94edce-8cdd-40dc-bd9e-68cc15000b6a" containerID="a8dd27139d5856a341f1bb226949fe5e5481e45d0c6fe16bebe5d91ef8310369" exitCode=255
Sep 18 19:15:28.929822 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.929791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv" event={"ID":"cc94edce-8cdd-40dc-bd9e-68cc15000b6a","Type":"ContainerDied","Data":"a8dd27139d5856a341f1bb226949fe5e5481e45d0c6fe16bebe5d91ef8310369"}
Sep 18 19:15:28.929899 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.929873    2559 scope.go:117] "RemoveContainer" containerID="5ec228d7a5e9c0eca119e6eb4c75a548695d775fb682d796ab2f973b5260baa4"
Sep 18 19:15:28.930160 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.930134    2559 scope.go:117] "RemoveContainer" containerID="a8dd27139d5856a341f1bb226949fe5e5481e45d0c6fe16bebe5d91ef8310369"
Sep 18 19:15:28.930377 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:28.930352    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-b9dcv_openshift-console-operator(cc94edce-8cdd-40dc-bd9e-68cc15000b6a)\"" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv" podUID="cc94edce-8cdd-40dc-bd9e-68cc15000b6a"
Sep 18 19:15:28.956893 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.956845    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-kfskl" podStartSLOduration=1.982902234 podStartE2EDuration="4.95682722s" totalImagesPullingTime="2.973924986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:27.983898899 +0000 UTC m=+50.438037076" watchObservedRunningTime="2026-09-18 19:15:28.95682722 +0000 UTC m=+51.410965405"
Sep 18 19:15:28.970680 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:28.970634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:28.970823 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:28.970804    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:28.970884 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:28.970873    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls podName:fbe654a7-76ea-4741-ae34-d2b4566cf622 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:44.970853953 +0000 UTC m=+67.424992120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls") pod "dns-default-fqqh8" (UID: "fbe654a7-76ea-4741-ae34-d2b4566cf622") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:29.127358 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.127328    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7pvzt"]
Sep 18 19:15:29.129928 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.129894    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.136044 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.136011    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:15:29.136415 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.136387    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wlcxl\""
Sep 18 19:15:29.136526 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.136499    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:15:29.157469 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.157446    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7pvzt"]
Sep 18 19:15:29.180525 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.180502    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf"]
Sep 18 19:15:29.185537 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.182500    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf"
Sep 18 19:15:29.186329 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.186304    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hgqs7\""
Sep 18 19:15:29.186443 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.186339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 19:15:29.186443 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.186428    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 19:15:29.196986 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.196965    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf"]
Sep 18 19:15:29.274052 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.274021    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-data-volume\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.274052 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.274052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-crio-socket\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.274262 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.274097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.274262 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.274157    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdk9\" (UniqueName: \"kubernetes.io/projected/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-api-access-qrdk9\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.274262 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.274244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.375396 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.375368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:29.375396 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.375397    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frf8\" (UniqueName: \"kubernetes.io/projected/59dee8c0-41d0-414e-b17b-99f7522088f9-kube-api-access-8frf8\") pod \"migrator-76794cbd6c-sktcf\" (UID: \"59dee8c0-41d0-414e-b17b-99f7522088f9\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf"
Sep 18 19:15:29.375571 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.375424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.375571 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.375485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdk9\" (UniqueName: \"kubernetes.io/projected/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-api-access-qrdk9\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.375571 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.375517    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:29.375571 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.375539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.375571 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.375562    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:29.875549368 +0000 UTC m=+52.329687522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:29.375756 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.375569    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:29.375756 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.375582    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:29.375756 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.375627    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert podName:048b4230-fb95-4853-8eaf-830198c66298 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:31.37560955 +0000 UTC m=+53.829747722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-bqhhx" (UID: "048b4230-fb95-4853-8eaf-830198c66298") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:29.375756 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.375646    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:29.875637232 +0000 UTC m=+52.329775402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:29.375756 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.375712    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-data-volume\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.375756 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.375740    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-crio-socket\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.375954 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.375924    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-crio-socket\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.375990 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.375978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-data-volume\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.385288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.385232    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdk9\" (UniqueName: \"kubernetes.io/projected/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-api-access-qrdk9\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.476296 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.476266    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frf8\" (UniqueName: \"kubernetes.io/projected/59dee8c0-41d0-414e-b17b-99f7522088f9-kube-api-access-8frf8\") pod \"migrator-76794cbd6c-sktcf\" (UID: \"59dee8c0-41d0-414e-b17b-99f7522088f9\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf"
Sep 18 19:15:29.485719 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.485700    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frf8\" (UniqueName: \"kubernetes.io/projected/59dee8c0-41d0-414e-b17b-99f7522088f9-kube-api-access-8frf8\") pod \"migrator-76794cbd6c-sktcf\" (UID: \"59dee8c0-41d0-414e-b17b-99f7522088f9\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf"
Sep 18 19:15:29.493575 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.493555    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf"
Sep 18 19:15:29.763719 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.763689    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf"]
Sep 18 19:15:29.765892 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:29.765867    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dee8c0_41d0_414e_b17b_99f7522088f9.slice/crio-d77e1cc0f245790da2cd673b5648e205d8cf4b4e3584c824f373107bf8dcc82a WatchSource:0}: Error finding container d77e1cc0f245790da2cd673b5648e205d8cf4b4e3584c824f373107bf8dcc82a: Status 404 returned error can't find the container with id d77e1cc0f245790da2cd673b5648e205d8cf4b4e3584c824f373107bf8dcc82a
Sep 18 19:15:29.879186 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.879156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.879313 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.879261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:29.879362 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.879315    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:29.879397 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.879378    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:30.879362268 +0000 UTC m=+53.333500441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:29.879397 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.879381    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:29.879466 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.879435    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:30.879421492 +0000 UTC m=+53.333559667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:29.933626 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.933601    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:15:29.933961 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.933935    2559 scope.go:117] "RemoveContainer" containerID="a8dd27139d5856a341f1bb226949fe5e5481e45d0c6fe16bebe5d91ef8310369"
Sep 18 19:15:29.934113 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:29.934085    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-b9dcv_openshift-console-operator(cc94edce-8cdd-40dc-bd9e-68cc15000b6a)\"" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv" podUID="cc94edce-8cdd-40dc-bd9e-68cc15000b6a"
Sep 18 19:15:29.934918 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:29.934897    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf" event={"ID":"59dee8c0-41d0-414e-b17b-99f7522088f9","Type":"ContainerStarted","Data":"d77e1cc0f245790da2cd673b5648e205d8cf4b4e3584c824f373107bf8dcc82a"}
Sep 18 19:15:30.830086 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:30.830055    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-67hqf_af4fccc9-2cf5-42f5-aa77-d7bc375a4c49/dns-node-resolver/0.log"
Sep 18 19:15:30.885473 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:30.885435    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:30.885642 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:30.885552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:30.885642 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:30.885602    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:30.885753 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:30.885658    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:30.885753 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:30.885682    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:32.885660808 +0000 UTC m=+55.339798986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:30.885753 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:30.885721    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:32.885704155 +0000 UTC m=+55.339842333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:31.388620 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.388586    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:31.388986 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:31.388762    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:31.388986 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:31.388841    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert podName:048b4230-fb95-4853-8eaf-830198c66298 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:35.388818831 +0000 UTC m=+57.842956990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-bqhhx" (UID: "048b4230-fb95-4853-8eaf-830198c66298") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:31.739902 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.739826    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-k2s8t"]
Sep 18 19:15:31.759937 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.759908    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-k2s8t"]
Sep 18 19:15:31.760055 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.760009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:31.763030 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.763008    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 19:15:31.763149 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.763027    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 19:15:31.763149 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.763105    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ctlqk\""
Sep 18 19:15:31.763804 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.763788    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 19:15:31.763883 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.763836    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 19:15:31.791755 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.791729    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95d723d4-3e56-4007-b497-1a62c8d50554-signing-cabundle\") pod \"service-ca-55f6fc7b86-k2s8t\" (UID: \"95d723d4-3e56-4007-b497-1a62c8d50554\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:31.791848 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.791756    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95d723d4-3e56-4007-b497-1a62c8d50554-signing-key\") pod \"service-ca-55f6fc7b86-k2s8t\" (UID: \"95d723d4-3e56-4007-b497-1a62c8d50554\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:31.791894 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.791846    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99k4n\" (UniqueName: \"kubernetes.io/projected/95d723d4-3e56-4007-b497-1a62c8d50554-kube-api-access-99k4n\") pod \"service-ca-55f6fc7b86-k2s8t\" (UID: \"95d723d4-3e56-4007-b497-1a62c8d50554\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:31.892598 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.892562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95d723d4-3e56-4007-b497-1a62c8d50554-signing-cabundle\") pod \"service-ca-55f6fc7b86-k2s8t\" (UID: \"95d723d4-3e56-4007-b497-1a62c8d50554\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:31.892598 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.892597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95d723d4-3e56-4007-b497-1a62c8d50554-signing-key\") pod \"service-ca-55f6fc7b86-k2s8t\" (UID: \"95d723d4-3e56-4007-b497-1a62c8d50554\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:31.892801 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.892756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99k4n\" (UniqueName: \"kubernetes.io/projected/95d723d4-3e56-4007-b497-1a62c8d50554-kube-api-access-99k4n\") pod \"service-ca-55f6fc7b86-k2s8t\" (UID: \"95d723d4-3e56-4007-b497-1a62c8d50554\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:31.893182 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.893162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95d723d4-3e56-4007-b497-1a62c8d50554-signing-cabundle\") pod \"service-ca-55f6fc7b86-k2s8t\" (UID: \"95d723d4-3e56-4007-b497-1a62c8d50554\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:31.894991 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.894970    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95d723d4-3e56-4007-b497-1a62c8d50554-signing-key\") pod \"service-ca-55f6fc7b86-k2s8t\" (UID: \"95d723d4-3e56-4007-b497-1a62c8d50554\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:31.901520 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.901499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99k4n\" (UniqueName: \"kubernetes.io/projected/95d723d4-3e56-4007-b497-1a62c8d50554-kube-api-access-99k4n\") pod \"service-ca-55f6fc7b86-k2s8t\" (UID: \"95d723d4-3e56-4007-b497-1a62c8d50554\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:31.942978 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.942946    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf" event={"ID":"59dee8c0-41d0-414e-b17b-99f7522088f9","Type":"ContainerStarted","Data":"062eb220af680c5dbfb408ecce2d165667dd8f5fdd85317fdefc61f58d2aa7f8"}
Sep 18 19:15:31.943100 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.942993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf" event={"ID":"59dee8c0-41d0-414e-b17b-99f7522088f9","Type":"ContainerStarted","Data":"e2f48eeaec4ff6937f65aaca2deb604835f331078bdf0eed985947c5383167b0"}
Sep 18 19:15:31.962061 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:31.961914    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-sktcf" podStartSLOduration=1.361107496 podStartE2EDuration="2.961900497s" totalImagesPullingTime="1.600793001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:31.961492702 +0000 UTC m=+54.415630878" watchObservedRunningTime="2026-09-18 19:15:31.961900497 +0000 UTC m=+54.416038674"
Sep 18 19:15:32.068167 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:32.068138    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t"
Sep 18 19:15:32.094621 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:32.094581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:32.094753 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:32.094701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:32.094821 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:32.094760    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:40.094741523 +0000 UTC m=+62.548879693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:32.094885 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:32.094829    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:32.094885 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:32.094874    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:40.094864753 +0000 UTC m=+62.549002909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : secret "router-metrics-certs-default" not found
Sep 18 19:15:32.215479 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:32.215456    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xfkrz_76d77ab0-46a5-432c-a812-67a2e57e0f07/node-ca/0.log"
Sep 18 19:15:32.356784 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:32.356708    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-k2s8t"]
Sep 18 19:15:32.359312 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:32.359281    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d723d4_3e56_4007_b497_1a62c8d50554.slice/crio-455e38769603e13f4d95a32828bed11dcc274a892ce0ab0516e9a301dd76a91b WatchSource:0}: Error finding container 455e38769603e13f4d95a32828bed11dcc274a892ce0ab0516e9a301dd76a91b: Status 404 returned error can't find the container with id 455e38769603e13f4d95a32828bed11dcc274a892ce0ab0516e9a301dd76a91b
Sep 18 19:15:32.817070 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:32.817026    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vq6nl_4af3fc5f-2070-42ee-a588-c4197598b28a/serve-healthcheck-canary/0.log"
Sep 18 19:15:32.900613 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:32.900576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:32.900740 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:32.900676    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:32.900801 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:32.900743    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:36.900728252 +0000 UTC m=+59.354866427 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:32.900846 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:32.900799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:32.900960 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:32.900943    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:32.900999 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:32.900993    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:36.900982282 +0000 UTC m=+59.355120437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:32.948164 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:32.948136    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t" event={"ID":"95d723d4-3e56-4007-b497-1a62c8d50554","Type":"ContainerStarted","Data":"455e38769603e13f4d95a32828bed11dcc274a892ce0ab0516e9a301dd76a91b"}
Sep 18 19:15:34.838362 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:34.838329    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:34.838758 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:34.838419    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:34.838758 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:34.838749    2559 scope.go:117] "RemoveContainer" containerID="a8dd27139d5856a341f1bb226949fe5e5481e45d0c6fe16bebe5d91ef8310369"
Sep 18 19:15:34.838912 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:34.838891    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-b9dcv_openshift-console-operator(cc94edce-8cdd-40dc-bd9e-68cc15000b6a)\"" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv" podUID="cc94edce-8cdd-40dc-bd9e-68cc15000b6a"
Sep 18 19:15:34.962395 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:34.962356    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t" event={"ID":"95d723d4-3e56-4007-b497-1a62c8d50554","Type":"ContainerStarted","Data":"96c8e5a2162867c7b23e2d2ead7c14d8fc7fb6614f92e3b53ce45f71bcfd8321"}
Sep 18 19:15:34.988044 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:34.987943    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-k2s8t" podStartSLOduration=1.926931489 podStartE2EDuration="3.987925619s" totalImagesPullingTime="2.06099413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:34.987113436 +0000 UTC m=+57.441251624" watchObservedRunningTime="2026-09-18 19:15:34.987925619 +0000 UTC m=+57.442063797"
Sep 18 19:15:35.419919 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:35.419881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:35.420121 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:35.419994    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:35.420121 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:35.420057    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert podName:048b4230-fb95-4853-8eaf-830198c66298 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:43.420042298 +0000 UTC m=+65.874180453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-bqhhx" (UID: "048b4230-fb95-4853-8eaf-830198c66298") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:36.930826 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:36.930793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:36.931188 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:36.930856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:36.931188 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:36.930943    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:36.931188 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:36.930945    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:36.931188 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:36.931003    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:44.930986339 +0000 UTC m=+67.385124494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:36.931188 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:36.931016    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:44.931010153 +0000 UTC m=+67.385148307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:38.861058 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:38.861031    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hfmt"
Sep 18 19:15:40.155814 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:40.155778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:40.156245 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:40.155884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:40.156245 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:40.155971    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:56.155944344 +0000 UTC m=+78.610082516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:40.156245 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:40.155993    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:40.156245 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:40.156039    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs podName:51a4f6b3-247d-4614-976d-e585479998d7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:56.156026703 +0000 UTC m=+78.610164858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs") pod "router-default-58ccb95cb-jb24t" (UID: "51a4f6b3-247d-4614-976d-e585479998d7") : secret "router-metrics-certs-default" not found
Sep 18 19:15:43.486570 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:43.486527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:43.486958 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:43.486683    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:43.486958 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:43.486756    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert podName:048b4230-fb95-4853-8eaf-830198c66298 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:59.486740534 +0000 UTC m=+81.940878689 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-bqhhx" (UID: "048b4230-fb95-4853-8eaf-830198c66298") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:44.999094 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:44.999059    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:44.999094 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:44.999094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:44.999513 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:44.999144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:44.999513 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:44.999213    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:44.999513 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:44.999278    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls podName:805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:00.999262384 +0000 UTC m=+83.453400558 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pvzt" (UID: "805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:44.999758 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:44.999738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:15:45.001456 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.001433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe654a7-76ea-4741-ae34-d2b4566cf622-metrics-tls\") pod \"dns-default-fqqh8\" (UID: \"fbe654a7-76ea-4741-ae34-d2b4566cf622\") " pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:45.195884 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.195853    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sd5f2\""
Sep 18 19:15:45.203757 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.203737    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:45.403302 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.403275    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:45.403452 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.403315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:45.406282 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.406266    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:15:45.407584 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.407568    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:15:45.416001 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.415972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edf078c1-1bad-4f23-8753-bd4414735c17-original-pull-secret\") pod \"global-pull-secret-syncer-pxfxm\" (UID: \"edf078c1-1bad-4f23-8753-bd4414735c17\") " pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:45.416097 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.416016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb350aae-48d3-4913-aec6-cd5f71e51350-metrics-certs\") pod \"network-metrics-daemon-g9cck\" (UID: \"cb350aae-48d3-4913-aec6-cd5f71e51350\") " pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:45.504354 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.504325    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rv2\" (UniqueName: \"kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2\") pod \"network-check-target-5hs2m\" (UID: \"bf8dd3b4-3f67-4029-b116-ee69dcc738c6\") " pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:45.507279 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.507262    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:15:45.519310 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.519288    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:15:45.527553 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.527532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rv2\" (UniqueName: \"kubernetes.io/projected/bf8dd3b4-3f67-4029-b116-ee69dcc738c6-kube-api-access-45rv2\") pod \"network-check-target-5hs2m\" (UID: \"bf8dd3b4-3f67-4029-b116-ee69dcc738c6\") " pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:45.535230 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.535168    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fqqh8"]
Sep 18 19:15:45.537965 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:45.537944    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe654a7_76ea_4741_ae34_d2b4566cf622.slice/crio-885d8c6fa75a3d99f5a9c593055946ea19717cedb6121ed419befc46fe6a351a WatchSource:0}: Error finding container 885d8c6fa75a3d99f5a9c593055946ea19717cedb6121ed419befc46fe6a351a: Status 404 returned error can't find the container with id 885d8c6fa75a3d99f5a9c593055946ea19717cedb6121ed419befc46fe6a351a
Sep 18 19:15:45.614589 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.614558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxfxm"
Sep 18 19:15:45.627331 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.627311    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-95w6x\""
Sep 18 19:15:45.633455 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.633438    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9cck"
Sep 18 19:15:45.635067 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.635053    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lf5lk\""
Sep 18 19:15:45.637804 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.637791    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:45.926240 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.926216    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pxfxm"]
Sep 18 19:15:45.926971 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:45.926945    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf078c1_1bad_4f23_8753_bd4414735c17.slice/crio-47ab4605de3526576af48ad95d73b8c1221a034b3880d5d28468f6ae7d941552 WatchSource:0}: Error finding container 47ab4605de3526576af48ad95d73b8c1221a034b3880d5d28468f6ae7d941552: Status 404 returned error can't find the container with id 47ab4605de3526576af48ad95d73b8c1221a034b3880d5d28468f6ae7d941552
Sep 18 19:15:45.979668 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.979643    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g9cck"]
Sep 18 19:15:45.983602 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:45.983577    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb350aae_48d3_4913_aec6_cd5f71e51350.slice/crio-4bb5f11c29b3de743e7a543bcae0411c78926acb9f5e42320d59a3412fdb8665 WatchSource:0}: Error finding container 4bb5f11c29b3de743e7a543bcae0411c78926acb9f5e42320d59a3412fdb8665: Status 404 returned error can't find the container with id 4bb5f11c29b3de743e7a543bcae0411c78926acb9f5e42320d59a3412fdb8665
Sep 18 19:15:45.995068 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.995039    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g9cck" event={"ID":"cb350aae-48d3-4913-aec6-cd5f71e51350","Type":"ContainerStarted","Data":"4bb5f11c29b3de743e7a543bcae0411c78926acb9f5e42320d59a3412fdb8665"}
Sep 18 19:15:45.996311 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.996284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pxfxm" event={"ID":"edf078c1-1bad-4f23-8753-bd4414735c17","Type":"ContainerStarted","Data":"47ab4605de3526576af48ad95d73b8c1221a034b3880d5d28468f6ae7d941552"}
Sep 18 19:15:45.997390 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:45.997370    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fqqh8" event={"ID":"fbe654a7-76ea-4741-ae34-d2b4566cf622","Type":"ContainerStarted","Data":"885d8c6fa75a3d99f5a9c593055946ea19717cedb6121ed419befc46fe6a351a"}
Sep 18 19:15:46.036792 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:46.036768    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8dd3b4_3f67_4029_b116_ee69dcc738c6.slice/crio-ecef1bfbaaabe376da1291729aac334a7b5ab586195e0f132c803c0bc7d705c0 WatchSource:0}: Error finding container ecef1bfbaaabe376da1291729aac334a7b5ab586195e0f132c803c0bc7d705c0: Status 404 returned error can't find the container with id ecef1bfbaaabe376da1291729aac334a7b5ab586195e0f132c803c0bc7d705c0
Sep 18 19:15:46.037772 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:46.037686    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5hs2m"]
Sep 18 19:15:47.004149 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:47.004109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5hs2m" event={"ID":"bf8dd3b4-3f67-4029-b116-ee69dcc738c6","Type":"ContainerStarted","Data":"ecef1bfbaaabe376da1291729aac334a7b5ab586195e0f132c803c0bc7d705c0"}
Sep 18 19:15:48.013513 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:48.013466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fqqh8" event={"ID":"fbe654a7-76ea-4741-ae34-d2b4566cf622","Type":"ContainerStarted","Data":"f42b14fa0075208fe48ea08b36026153ceea2ca2f1e1352efaaf97bb9ddc5089"}
Sep 18 19:15:48.017022 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:48.016989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g9cck" event={"ID":"cb350aae-48d3-4913-aec6-cd5f71e51350","Type":"ContainerStarted","Data":"5ef32f187a4f81bca9908436c96a6928ebc72e0686696ea174e9048f7d8dd356"}
Sep 18 19:15:49.023302 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:49.023236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g9cck" event={"ID":"cb350aae-48d3-4913-aec6-cd5f71e51350","Type":"ContainerStarted","Data":"52f6285d4261e0ae2a703ee982e90371bde413e0a9e476af7278e4f7b83fc6b9"}
Sep 18 19:15:49.025707 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:49.025652    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fqqh8" event={"ID":"fbe654a7-76ea-4741-ae34-d2b4566cf622","Type":"ContainerStarted","Data":"22786dc4d0994fed4e31629fe8d8d5437e6b1f6b80ebe273d62ee22d5f4e8817"}
Sep 18 19:15:49.025828 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:49.025779    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:49.081612 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:49.081565    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fqqh8" podStartSLOduration=33.803807698 podStartE2EDuration="36.081549589s" totalImagesPullingTime="2.277741891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:49.079233258 +0000 UTC m=+71.533371435" watchObservedRunningTime="2026-09-18 19:15:49.081549589 +0000 UTC m=+71.535687793"
Sep 18 19:15:49.082196 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:49.082164    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g9cck" podStartSLOduration=67.25150879 podStartE2EDuration="1m9.082156619s" totalImagesPullingTime="1.830647829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:49.049996768 +0000 UTC m=+71.504134945" watchObservedRunningTime="2026-09-18 19:15:49.082156619 +0000 UTC m=+71.536294795"
Sep 18 19:15:50.705505 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:50.705482    2559 scope.go:117] "RemoveContainer" containerID="a8dd27139d5856a341f1bb226949fe5e5481e45d0c6fe16bebe5d91ef8310369"
Sep 18 19:15:51.034148 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:51.034114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pxfxm" event={"ID":"edf078c1-1bad-4f23-8753-bd4414735c17","Type":"ContainerStarted","Data":"de79aa772b9a7ae90fb9659136da424f493b2e4aaef915882bb453236e0a4b7d"}
Sep 18 19:15:51.035751 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:51.035725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5hs2m" event={"ID":"bf8dd3b4-3f67-4029-b116-ee69dcc738c6","Type":"ContainerStarted","Data":"f4926bea66000f6fb2a042dda3ebab8b7a0d61f6b633217556e3736d48f7b8c5"}
Sep 18 19:15:51.035883 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:51.035829    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:15:51.037625 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:51.037609    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:15:51.037701 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:51.037682    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv" event={"ID":"cc94edce-8cdd-40dc-bd9e-68cc15000b6a","Type":"ContainerStarted","Data":"c5e991118bd041094c794003fa5f389669f26b7be5a00f7b0a25f273d13eb0e8"}
Sep 18 19:15:51.037945 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:51.037925    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:51.060029 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:51.059981    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pxfxm" podStartSLOduration=66.355234546 podStartE2EDuration="1m11.059966968s" totalImagesPullingTime="4.704732422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:51.059481026 +0000 UTC m=+73.513619204" watchObservedRunningTime="2026-09-18 19:15:51.059966968 +0000 UTC m=+73.514105148"
Sep 18 19:15:51.090890 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:51.090846    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5hs2m" podStartSLOduration=67.500162327 podStartE2EDuration="1m12.090836399s" totalImagesPullingTime="4.590674072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:51.089944028 +0000 UTC m=+73.544082204" watchObservedRunningTime="2026-09-18 19:15:51.090836399 +0000 UTC m=+73.544974587"
Sep 18 19:15:51.119871 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:51.119834    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv" podStartSLOduration=24.485796216 podStartE2EDuration="27.11982402s" totalImagesPullingTime="2.634027804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:51.119272809 +0000 UTC m=+73.573410985" watchObservedRunningTime="2026-09-18 19:15:51.11982402 +0000 UTC m=+73.573962197"
Sep 18 19:15:51.276099 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:51.276073    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-b9dcv"
Sep 18 19:15:56.200552 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:56.200509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:56.201019 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:56.200597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:56.201218 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:56.201176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a4f6b3-247d-4614-976d-e585479998d7-service-ca-bundle\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:56.202981 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:56.202960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a4f6b3-247d-4614-976d-e585479998d7-metrics-certs\") pod \"router-default-58ccb95cb-jb24t\" (UID: \"51a4f6b3-247d-4614-976d-e585479998d7\") " pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:56.403139 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:56.403110    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8nw9x\""
Sep 18 19:15:56.408894 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:56.408880    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:56.696845 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:56.696817    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58ccb95cb-jb24t"]
Sep 18 19:15:56.699812 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:15:56.699785    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a4f6b3_247d_4614_976d_e585479998d7.slice/crio-e91014d3694f422fa29ac01a49ab6cb7e93ba341bc6bb8746aaeddf82653f1cd WatchSource:0}: Error finding container e91014d3694f422fa29ac01a49ab6cb7e93ba341bc6bb8746aaeddf82653f1cd: Status 404 returned error can't find the container with id e91014d3694f422fa29ac01a49ab6cb7e93ba341bc6bb8746aaeddf82653f1cd
Sep 18 19:15:57.059136 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:57.059105    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58ccb95cb-jb24t" event={"ID":"51a4f6b3-247d-4614-976d-e585479998d7","Type":"ContainerStarted","Data":"12614d164ebf539b27fe992bb47ad23023b88a2ddcc1fc3aebfc89306dcdaa3a"}
Sep 18 19:15:57.059307 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:57.059152    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58ccb95cb-jb24t" event={"ID":"51a4f6b3-247d-4614-976d-e585479998d7","Type":"ContainerStarted","Data":"e91014d3694f422fa29ac01a49ab6cb7e93ba341bc6bb8746aaeddf82653f1cd"}
Sep 18 19:15:57.110681 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:57.110635    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-58ccb95cb-jb24t" podStartSLOduration=33.110619051 podStartE2EDuration="33.110619051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:57.110586462 +0000 UTC m=+79.564724640" watchObservedRunningTime="2026-09-18 19:15:57.110619051 +0000 UTC m=+79.564757269"
Sep 18 19:15:57.409317 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:57.409238    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:57.411774 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:57.411750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:58.062345 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:58.062307    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:58.063582 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:58.063561    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-58ccb95cb-jb24t"
Sep 18 19:15:59.031469 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.031438    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fqqh8"
Sep 18 19:15:59.316703 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.316617    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"]
Sep 18 19:15:59.351797 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.351770    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"]
Sep 18 19:15:59.351958 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.351859    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"
Sep 18 19:15:59.355269 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.355245    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 19:15:59.355400 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.355329    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h9m9v\""
Sep 18 19:15:59.425373 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.425345    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/caf43189-4ad3-42c5-81c2-580ec4ed8956-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7lghg\" (UID: \"caf43189-4ad3-42c5-81c2-580ec4ed8956\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"
Sep 18 19:15:59.526221 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.526156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/caf43189-4ad3-42c5-81c2-580ec4ed8956-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7lghg\" (UID: \"caf43189-4ad3-42c5-81c2-580ec4ed8956\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"
Sep 18 19:15:59.526409 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.526273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:59.526409 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:59.526321    2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 18 19:15:59.526409 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:15:59.526388    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf43189-4ad3-42c5-81c2-580ec4ed8956-tls-certificates podName:caf43189-4ad3-42c5-81c2-580ec4ed8956 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:00.026371452 +0000 UTC m=+82.480509607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/caf43189-4ad3-42c5-81c2-580ec4ed8956-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-7lghg" (UID: "caf43189-4ad3-42c5-81c2-580ec4ed8956") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 18 19:15:59.528703 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.528684    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/048b4230-fb95-4853-8eaf-830198c66298-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-bqhhx\" (UID: \"048b4230-fb95-4853-8eaf-830198c66298\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:15:59.709987 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.709915    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tmph6\""
Sep 18 19:15:59.717648 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:15:59.717626    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"
Sep 18 19:16:00.013254 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:00.013224    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx"]
Sep 18 19:16:00.015104 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:16:00.015079    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048b4230_fb95_4853_8eaf_830198c66298.slice/crio-4e278aef1ad06eab5c6982329774ed2d3b4a2d3d470489128dba4b9002d3015f WatchSource:0}: Error finding container 4e278aef1ad06eab5c6982329774ed2d3b4a2d3d470489128dba4b9002d3015f: Status 404 returned error can't find the container with id 4e278aef1ad06eab5c6982329774ed2d3b4a2d3d470489128dba4b9002d3015f
Sep 18 19:16:00.030130 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:00.030108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/caf43189-4ad3-42c5-81c2-580ec4ed8956-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7lghg\" (UID: \"caf43189-4ad3-42c5-81c2-580ec4ed8956\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"
Sep 18 19:16:00.032431 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:00.032410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/caf43189-4ad3-42c5-81c2-580ec4ed8956-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7lghg\" (UID: \"caf43189-4ad3-42c5-81c2-580ec4ed8956\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"
Sep 18 19:16:00.069960 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:00.069932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx" event={"ID":"048b4230-fb95-4853-8eaf-830198c66298","Type":"ContainerStarted","Data":"4e278aef1ad06eab5c6982329774ed2d3b4a2d3d470489128dba4b9002d3015f"}
Sep 18 19:16:00.262534 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:00.262460    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"
Sep 18 19:16:00.536557 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:00.536535    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"]
Sep 18 19:16:00.539215 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:16:00.539172    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf43189_4ad3_42c5_81c2_580ec4ed8956.slice/crio-f1c4ac094e77859540f1573b23ca233e63bb066b313b81fe02f6630511d0cc0a WatchSource:0}: Error finding container f1c4ac094e77859540f1573b23ca233e63bb066b313b81fe02f6630511d0cc0a: Status 404 returned error can't find the container with id f1c4ac094e77859540f1573b23ca233e63bb066b313b81fe02f6630511d0cc0a
Sep 18 19:16:01.036046 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:01.036012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:16:01.038660 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:01.038626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pvzt\" (UID: \"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2\") " pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:16:01.075393 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:01.075359    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg" event={"ID":"caf43189-4ad3-42c5-81c2-580ec4ed8956","Type":"ContainerStarted","Data":"f1c4ac094e77859540f1573b23ca233e63bb066b313b81fe02f6630511d0cc0a"}
Sep 18 19:16:01.243302 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:01.243268    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wlcxl\""
Sep 18 19:16:01.249954 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:01.249928    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7pvzt"
Sep 18 19:16:01.538069 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:01.538039    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7pvzt"]
Sep 18 19:16:01.708417 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:16:01.708355    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805bdd1c_3ae6_4b11_ac88_ed2fc7af00a2.slice/crio-10b5553eba11e633f73bd94e63fb211f4616200248fab34aee42a5c92501bd22 WatchSource:0}: Error finding container 10b5553eba11e633f73bd94e63fb211f4616200248fab34aee42a5c92501bd22: Status 404 returned error can't find the container with id 10b5553eba11e633f73bd94e63fb211f4616200248fab34aee42a5c92501bd22
Sep 18 19:16:02.080373 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:02.080339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7pvzt" event={"ID":"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2","Type":"ContainerStarted","Data":"81466add5c19613e08e6eb8432489a6f80ab110477871b84b72236dca071d156"}
Sep 18 19:16:02.080739 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:02.080398    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7pvzt" event={"ID":"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2","Type":"ContainerStarted","Data":"10b5553eba11e633f73bd94e63fb211f4616200248fab34aee42a5c92501bd22"}
Sep 18 19:16:02.084103 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:02.084069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx" event={"ID":"048b4230-fb95-4853-8eaf-830198c66298","Type":"ContainerStarted","Data":"e37ee50cdf4ade00649c6cfd1b57f54ad14e703d7e1992a4319ed78ad1da9980"}
Sep 18 19:16:02.085704 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:02.085680    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg" event={"ID":"caf43189-4ad3-42c5-81c2-580ec4ed8956","Type":"ContainerStarted","Data":"d605919f087e413b745adaf9a7975c3210f40500f02896c30106acdb6c47bc00"}
Sep 18 19:16:02.085922 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:02.085904    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"
Sep 18 19:16:02.090451 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:02.090432    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg"
Sep 18 19:16:02.104948 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:02.104906    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-bqhhx" podStartSLOduration=34.025297086 podStartE2EDuration="35.104892308s" totalImagesPullingTime="1.079595222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:02.103997652 +0000 UTC m=+84.558135830" watchObservedRunningTime="2026-09-18 19:16:02.104892308 +0000 UTC m=+84.559030515"
Sep 18 19:16:02.125076 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:02.125035    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7lghg" podStartSLOduration=1.929919199 podStartE2EDuration="3.125021054s" totalImagesPullingTime="1.195101855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:02.124454488 +0000 UTC m=+84.578592664" watchObservedRunningTime="2026-09-18 19:16:02.125021054 +0000 UTC m=+84.579159222"
Sep 18 19:16:03.092393 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:03.091892    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7pvzt" event={"ID":"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2","Type":"ContainerStarted","Data":"dea242bddd94844ece062eb7dd3bd13c974e43dc43d4a1c762a1d654c67026aa"}
Sep 18 19:16:04.097714 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:04.097676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7pvzt" event={"ID":"805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2","Type":"ContainerStarted","Data":"07f92e68b940537f0e6e7aae3284ac857c280f83236e02e589a1fc414f807871"}
Sep 18 19:16:04.126138 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:04.126090    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7pvzt" podStartSLOduration=33.357968331 podStartE2EDuration="35.126074539s" totalImagesPullingTime="1.768106208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:15:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:04.124815144 +0000 UTC m=+86.578953324" watchObservedRunningTime="2026-09-18 19:16:04.126074539 +0000 UTC m=+86.580212716"
Sep 18 19:16:11.413747 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.413710    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85579c4694-sdcjp"]
Sep 18 19:16:11.418412 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.418388    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.423313 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.422452    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:16:11.423589 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.423563    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4hjjs\""
Sep 18 19:16:11.423711 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.423572    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:16:11.423711 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.423631    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:16:11.423711 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.423571    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:16:11.423711 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.423572    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:16:11.423921 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.423771    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:16:11.423921 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.423810    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:16:11.427701 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.427679    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 19:16:11.434936 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.434918    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85579c4694-sdcjp"]
Sep 18 19:16:11.514589 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.514531    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-oauth-config\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.514743 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.514598    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-service-ca\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.514743 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.514667    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-serving-cert\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.514743 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.514702    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-trusted-ca-bundle\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.514743 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.514725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-config\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.514879 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.514747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-oauth-serving-cert\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.514879 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.514774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczbr\" (UniqueName: \"kubernetes.io/projected/379405c9-7e19-4eeb-8eaa-8f547b20d410-kube-api-access-wczbr\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.615961 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.615924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-serving-cert\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.615961 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.615968    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-trusted-ca-bundle\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.616175 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.615986    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-config\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.616175 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.616003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-oauth-serving-cert\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.616175 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.616018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczbr\" (UniqueName: \"kubernetes.io/projected/379405c9-7e19-4eeb-8eaa-8f547b20d410-kube-api-access-wczbr\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.616175 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.616145    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-oauth-config\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.616387 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.616196    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-service-ca\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.616783 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.616757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-config\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.616952 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.616835    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-service-ca\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.616952 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.616861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-oauth-serving-cert\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.617125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.617107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-trusted-ca-bundle\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.618490 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.618468    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-serving-cert\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.618566 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.618501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-oauth-config\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.638823 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.638795    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczbr\" (UniqueName: \"kubernetes.io/projected/379405c9-7e19-4eeb-8eaa-8f547b20d410-kube-api-access-wczbr\") pod \"console-85579c4694-sdcjp\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") " pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:11.728989 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:11.728901    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:12.025883 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:12.025855    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85579c4694-sdcjp"]
Sep 18 19:16:12.028891 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:16:12.028860    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379405c9_7e19_4eeb_8eaa_8f547b20d410.slice/crio-ee73d110ebcec29b27fe26b22e530358c4cc140d54794f5f2d45271a4871b2f0 WatchSource:0}: Error finding container ee73d110ebcec29b27fe26b22e530358c4cc140d54794f5f2d45271a4871b2f0: Status 404 returned error can't find the container with id ee73d110ebcec29b27fe26b22e530358c4cc140d54794f5f2d45271a4871b2f0
Sep 18 19:16:12.125152 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:12.125121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85579c4694-sdcjp" event={"ID":"379405c9-7e19-4eeb-8eaa-8f547b20d410","Type":"ContainerStarted","Data":"ee73d110ebcec29b27fe26b22e530358c4cc140d54794f5f2d45271a4871b2f0"}
Sep 18 19:16:16.141649 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:16.141563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85579c4694-sdcjp" event={"ID":"379405c9-7e19-4eeb-8eaa-8f547b20d410","Type":"ContainerStarted","Data":"23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75"}
Sep 18 19:16:16.189295 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:16.189241    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85579c4694-sdcjp" podStartSLOduration=1.357206553 podStartE2EDuration="5.189226227s" totalImagesPullingTime="3.832019674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:16.188901589 +0000 UTC m=+98.643039767" watchObservedRunningTime="2026-09-18 19:16:16.189226227 +0000 UTC m=+98.643364395"
Sep 18 19:16:18.840253 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.840216    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2d9kz"]
Sep 18 19:16:18.844673 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.844653    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:18.856302 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.856281    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:16:18.856412 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.856282    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w8xns\""
Sep 18 19:16:18.856412 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.856282    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:16:18.856412 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.856282    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:16:18.856412 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.856281    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:16:18.857362 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.857348    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:16:18.857906 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.857891    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:16:18.972572 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.972544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e476d1f0-b191-403c-9e39-db73496b2834-metrics-client-ca\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:18.972572 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.972577    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmv2\" (UniqueName: \"kubernetes.io/projected/e476d1f0-b191-403c-9e39-db73496b2834-kube-api-access-xnmv2\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:18.972780 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.972596    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-accelerators-collector-config\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:18.972780 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.972614    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:18.972780 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.972727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-textfile\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:18.972780 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.972766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e476d1f0-b191-403c-9e39-db73496b2834-sys\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:18.972966 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.972791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-wtmp\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:18.972966 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.972861    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e476d1f0-b191-403c-9e39-db73496b2834-root\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:18.972966 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:18.972901    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-tls\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073313 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e476d1f0-b191-403c-9e39-db73496b2834-metrics-client-ca\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073471 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmv2\" (UniqueName: \"kubernetes.io/projected/e476d1f0-b191-403c-9e39-db73496b2834-kube-api-access-xnmv2\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073471 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-accelerators-collector-config\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073471 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073471 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-textfile\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073642 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e476d1f0-b191-403c-9e39-db73496b2834-sys\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073642 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-wtmp\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073642 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e476d1f0-b191-403c-9e39-db73496b2834-root\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073642 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-tls\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073642 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e476d1f0-b191-403c-9e39-db73496b2834-sys\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073854 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073664    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e476d1f0-b191-403c-9e39-db73496b2834-root\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073854 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073739    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-wtmp\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.073854 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-textfile\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.074011 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.073993    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e476d1f0-b191-403c-9e39-db73496b2834-metrics-client-ca\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.074079 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.074034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-accelerators-collector-config\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.075768 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.075748    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.075933 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.075916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e476d1f0-b191-403c-9e39-db73496b2834-node-exporter-tls\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.097953 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.097876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmv2\" (UniqueName: \"kubernetes.io/projected/e476d1f0-b191-403c-9e39-db73496b2834-kube-api-access-xnmv2\") pod \"node-exporter-2d9kz\" (UID: \"e476d1f0-b191-403c-9e39-db73496b2834\") " pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.153392 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:19.153366    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2d9kz"
Sep 18 19:16:19.165537 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:16:19.165508    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode476d1f0_b191_403c_9e39_db73496b2834.slice/crio-9ee60c8485a151081c00a2cf7da45aecf8a467288ee8baad6b9f82f642b1d39d WatchSource:0}: Error finding container 9ee60c8485a151081c00a2cf7da45aecf8a467288ee8baad6b9f82f642b1d39d: Status 404 returned error can't find the container with id 9ee60c8485a151081c00a2cf7da45aecf8a467288ee8baad6b9f82f642b1d39d
Sep 18 19:16:20.043389 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.043355    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:16:20.046997 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.046975    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.053440 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.053399    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 19:16:20.053440 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.053399    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 19:16:20.053440 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.053427    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 19:16:20.053658 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.053409    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 19:16:20.053658 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.053399    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 19:16:20.054017 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.053858    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 19:16:20.054115 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.054093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 19:16:20.054235 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.054219    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 19:16:20.054322 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.054299    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-626rt\""
Sep 18 19:16:20.068466 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.068444    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 19:16:20.116765 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.116738    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:16:20.157291 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.157257    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2d9kz" event={"ID":"e476d1f0-b191-403c-9e39-db73496b2834","Type":"ContainerStarted","Data":"37a0599b280408704ca17048be4e44a878e794571de6152f85ddf9cda7caab06"}
Sep 18 19:16:20.157388 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.157315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2d9kz" event={"ID":"e476d1f0-b191-403c-9e39-db73496b2834","Type":"ContainerStarted","Data":"9ee60c8485a151081c00a2cf7da45aecf8a467288ee8baad6b9f82f642b1d39d"}
Sep 18 19:16:20.182516 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182485    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-out\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.182675 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.182675 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.182675 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.182675 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182612    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.182675 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.182895 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182740    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.182895 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.182895 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-volume\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.183024 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.183024 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.182980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8597v\" (UniqueName: \"kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-kube-api-access-8597v\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.183099 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.183019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-web-config\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.183099 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.183066    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284091    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284306 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8597v\" (UniqueName: \"kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-kube-api-access-8597v\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284306 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284158    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-web-config\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284306 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284180    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284306 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-out\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284306 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284264    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284358    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284855 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-volume\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.284906 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.284862    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.285542 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.285320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.286189 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.285678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.287371 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.287345    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.287512 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.287482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.287990 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.287951    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-web-config\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.287990 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.287959    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-volume\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.288142 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.287988    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.288293 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.288275    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-out\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.288456 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.288427    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.288851 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.288832    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.289090 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.289075    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.304073 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.304024    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8597v\" (UniqueName: \"kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-kube-api-access-8597v\") pod \"alertmanager-main-0\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.356531 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.356499    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:20.650852 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:20.650782    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:16:20.653933 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:16:20.653908    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode435197f_98bb_4df0_8fc6_8cba4f3bf5e4.slice/crio-b1f7e224f767ef7f2fe7dc7d0f210852ca72bd20a5b4f5320cf11d1b7f6091e7 WatchSource:0}: Error finding container b1f7e224f767ef7f2fe7dc7d0f210852ca72bd20a5b4f5320cf11d1b7f6091e7: Status 404 returned error can't find the container with id b1f7e224f767ef7f2fe7dc7d0f210852ca72bd20a5b4f5320cf11d1b7f6091e7
Sep 18 19:16:21.162502 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:21.162462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerStarted","Data":"b1f7e224f767ef7f2fe7dc7d0f210852ca72bd20a5b4f5320cf11d1b7f6091e7"}
Sep 18 19:16:21.164344 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:21.164319    2559 generic.go:356] "Generic (PLEG): container finished" podID="e476d1f0-b191-403c-9e39-db73496b2834" containerID="37a0599b280408704ca17048be4e44a878e794571de6152f85ddf9cda7caab06" exitCode=0
Sep 18 19:16:21.164464 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:21.164357    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2d9kz" event={"ID":"e476d1f0-b191-403c-9e39-db73496b2834","Type":"ContainerDied","Data":"37a0599b280408704ca17048be4e44a878e794571de6152f85ddf9cda7caab06"}
Sep 18 19:16:21.729598 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:21.729567    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:21.729749 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:21.729613    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:21.734272 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:21.734252    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:22.043893 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.043872    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5hs2m"
Sep 18 19:16:22.169801 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.169772    2559 generic.go:356] "Generic (PLEG): container finished" podID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerID="5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51" exitCode=0
Sep 18 19:16:22.170178 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.169860    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerDied","Data":"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"}
Sep 18 19:16:22.172362 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.172343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2d9kz" event={"ID":"e476d1f0-b191-403c-9e39-db73496b2834","Type":"ContainerStarted","Data":"850bffd43b8e168b4fe362aba2bcbdf98985c5d1e2f4a7d53cd7572426d2e63e"}
Sep 18 19:16:22.172460 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.172392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2d9kz" event={"ID":"e476d1f0-b191-403c-9e39-db73496b2834","Type":"ContainerStarted","Data":"54b8c7f02ea699073222e533eb62847f108e8713c6e854f430d0848af36e0470"}
Sep 18 19:16:22.176270 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.176252    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:16:22.293300 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.293179    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2d9kz" podStartSLOduration=3.31467534 podStartE2EDuration="4.293161943s" totalImagesPullingTime="914.08671ms" totalInitContainerRuntime="64.399893ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:22.292047272 +0000 UTC m=+104.746185450" watchObservedRunningTime="2026-09-18 19:16:22.293161943 +0000 UTC m=+104.747300120"
Sep 18 19:16:22.696026 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.695948    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"]
Sep 18 19:16:22.712051 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.712022    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:22.726318 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.726290    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 19:16:22.727397 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.727371    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 19:16:22.727633 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.727609    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 19:16:22.727849 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.727832    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 19:16:22.728032 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.728017    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ne0dns1gdd81\""
Sep 18 19:16:22.729174 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.729025    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zffbp\""
Sep 18 19:16:22.729659 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.729635    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"]
Sep 18 19:16:22.731414 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.731393    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 19:16:22.909195 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.909158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-grpc-tls\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:22.909377 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.909222    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:22.909377 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.909284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:22.909377 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.909360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-tls\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:22.909521 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.909442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:22.909521 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.909495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff591b00-0c33-469a-9d64-c9aca37b5830-metrics-client-ca\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:22.909615 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.909525    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:22.909615 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:22.909546    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dr4\" (UniqueName: \"kubernetes.io/projected/ff591b00-0c33-469a-9d64-c9aca37b5830-kube-api-access-d7dr4\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.010428 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.010354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-grpc-tls\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.010428 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.010399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.010655 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.010438    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.010655 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.010493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-tls\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.010655 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.010523    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.010655 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.010563    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff591b00-0c33-469a-9d64-c9aca37b5830-metrics-client-ca\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.010655 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.010594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.010655 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.010629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dr4\" (UniqueName: \"kubernetes.io/projected/ff591b00-0c33-469a-9d64-c9aca37b5830-kube-api-access-d7dr4\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.011583 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.011551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff591b00-0c33-469a-9d64-c9aca37b5830-metrics-client-ca\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.013546 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.013509    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.013660 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.013626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-tls\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.013765 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.013741    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.014509 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.014484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-grpc-tls\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.014933 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.014881    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.015025 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.015001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff591b00-0c33-469a-9d64-c9aca37b5830-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.023848 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.023821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dr4\" (UniqueName: \"kubernetes.io/projected/ff591b00-0c33-469a-9d64-c9aca37b5830-kube-api-access-d7dr4\") pod \"thanos-querier-74478c4dc8-9s9s5\" (UID: \"ff591b00-0c33-469a-9d64-c9aca37b5830\") " pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.024004 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.023989    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:23.339095 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:23.339063    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"]
Sep 18 19:16:23.683146 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:16:23.683076    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff591b00_0c33_469a_9d64_c9aca37b5830.slice/crio-f894dbfa7ec20a8e61875ec391f807471123d126dc6063f1877fdebe974c9a0d WatchSource:0}: Error finding container f894dbfa7ec20a8e61875ec391f807471123d126dc6063f1877fdebe974c9a0d: Status 404 returned error can't find the container with id f894dbfa7ec20a8e61875ec391f807471123d126dc6063f1877fdebe974c9a0d
Sep 18 19:16:24.022974 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.022944    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78d5965bf4-9l5g9"]
Sep 18 19:16:24.043156 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.043135    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.046553 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.046531    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78d5965bf4-9l5g9"]
Sep 18 19:16:24.121297 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.121271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-trusted-ca-bundle\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.121415 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.121364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-service-ca\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.121415 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.121399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-oauth-config\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.121523 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.121446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-oauth-serving-cert\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.121523 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.121488    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9wjq\" (UniqueName: \"kubernetes.io/projected/741aae46-ad5c-498b-a312-46a0e643968e-kube-api-access-t9wjq\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.121523 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.121506    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-console-config\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.121680 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.121569    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-serving-cert\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.183198 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.183164    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerStarted","Data":"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"}
Sep 18 19:16:24.184500 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.184473    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5" event={"ID":"ff591b00-0c33-469a-9d64-c9aca37b5830","Type":"ContainerStarted","Data":"f894dbfa7ec20a8e61875ec391f807471123d126dc6063f1877fdebe974c9a0d"}
Sep 18 19:16:24.222077 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.222050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-oauth-serving-cert\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.222172 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.222109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9wjq\" (UniqueName: \"kubernetes.io/projected/741aae46-ad5c-498b-a312-46a0e643968e-kube-api-access-t9wjq\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.222172 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.222145    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-console-config\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.222317 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.222174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-serving-cert\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.222317 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.222230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-trusted-ca-bundle\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.222317 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.222288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-service-ca\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.222457 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.222315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-oauth-config\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.222944 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.222869    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-oauth-serving-cert\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.223377 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.223351    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-console-config\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.223558 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.223533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-service-ca\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.223831 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.223806    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-trusted-ca-bundle\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.224731 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.224712    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-oauth-config\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.224807 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.224748    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-serving-cert\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.235259 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.235234    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9wjq\" (UniqueName: \"kubernetes.io/projected/741aae46-ad5c-498b-a312-46a0e643968e-kube-api-access-t9wjq\") pod \"console-78d5965bf4-9l5g9\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") " pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.364460 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.364431    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:24.674017 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:24.673866    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78d5965bf4-9l5g9"]
Sep 18 19:16:24.676183 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:16:24.676142    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741aae46_ad5c_498b_a312_46a0e643968e.slice/crio-a5772925e402d1e010bb1ca459a929624a1f9b39ddde804d38b1d2f38fa82708 WatchSource:0}: Error finding container a5772925e402d1e010bb1ca459a929624a1f9b39ddde804d38b1d2f38fa82708: Status 404 returned error can't find the container with id a5772925e402d1e010bb1ca459a929624a1f9b39ddde804d38b1d2f38fa82708
Sep 18 19:16:25.194526 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:25.194487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerStarted","Data":"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"}
Sep 18 19:16:25.194674 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:25.194587    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerStarted","Data":"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"}
Sep 18 19:16:25.194674 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:25.194632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerStarted","Data":"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"}
Sep 18 19:16:25.194674 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:25.194672    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerStarted","Data":"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"}
Sep 18 19:16:25.196825 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:25.196761    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d5965bf4-9l5g9" event={"ID":"741aae46-ad5c-498b-a312-46a0e643968e","Type":"ContainerStarted","Data":"66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0"}
Sep 18 19:16:25.196825 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:25.196813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d5965bf4-9l5g9" event={"ID":"741aae46-ad5c-498b-a312-46a0e643968e","Type":"ContainerStarted","Data":"a5772925e402d1e010bb1ca459a929624a1f9b39ddde804d38b1d2f38fa82708"}
Sep 18 19:16:25.221782 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:25.221728    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78d5965bf4-9l5g9" podStartSLOduration=2.221705489 podStartE2EDuration="2.221705489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:25.219651007 +0000 UTC m=+107.673789185" watchObservedRunningTime="2026-09-18 19:16:25.221705489 +0000 UTC m=+107.675843667"
Sep 18 19:16:26.203354 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:26.203325    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5" event={"ID":"ff591b00-0c33-469a-9d64-c9aca37b5830","Type":"ContainerStarted","Data":"2f6dac4ef2f8fff1b005f75bc58836e7cc1dec7fa9e3990d828f16917c2e436d"}
Sep 18 19:16:26.203716 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:26.203416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5" event={"ID":"ff591b00-0c33-469a-9d64-c9aca37b5830","Type":"ContainerStarted","Data":"eb2bd4b0c61cd7a9bd45a2a02a6af4a5a91c5f43bc9e04bdd6cb0449f0a2c1f3"}
Sep 18 19:16:26.203716 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:26.203457    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5" event={"ID":"ff591b00-0c33-469a-9d64-c9aca37b5830","Type":"ContainerStarted","Data":"e863136d1e04998b3e565fe7cd403e8680ac05ee280d70350d3cd73ca8291401"}
Sep 18 19:16:26.206845 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:26.206814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerStarted","Data":"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"}
Sep 18 19:16:26.242762 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:26.242719    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.645241767 podStartE2EDuration="7.242704036s" totalImagesPullingTime="4.58444682s" totalInitContainerRuntime="13.015449ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:26.240232703 +0000 UTC m=+108.694370880" watchObservedRunningTime="2026-09-18 19:16:26.242704036 +0000 UTC m=+108.696842214"
Sep 18 19:16:27.214383 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:27.214339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5" event={"ID":"ff591b00-0c33-469a-9d64-c9aca37b5830","Type":"ContainerStarted","Data":"5dbbf949ba585afcf4c0798c0bb317a30c1c5334f62276b616ab5ce149684da7"}
Sep 18 19:16:27.214810 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:27.214409    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5" event={"ID":"ff591b00-0c33-469a-9d64-c9aca37b5830","Type":"ContainerStarted","Data":"c758fe268f44c892ddd64707778533d82383af01139a69b4dd043aa7ee906be6"}
Sep 18 19:16:27.214810 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:27.214439    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5" event={"ID":"ff591b00-0c33-469a-9d64-c9aca37b5830","Type":"ContainerStarted","Data":"d7c196da639270855a907d2eb093459b7a2f11440cb53debd62e6fa70a5b2b7b"}
Sep 18 19:16:27.214810 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:27.214803    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:27.241434 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:27.241371    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5" podStartSLOduration=2.907824861 podStartE2EDuration="5.241358391s" totalImagesPullingTime="2.33353353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:27.240016432 +0000 UTC m=+109.694154609" watchObservedRunningTime="2026-09-18 19:16:27.241358391 +0000 UTC m=+109.695496584"
Sep 18 19:16:33.225838 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:33.225811    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74478c4dc8-9s9s5"
Sep 18 19:16:34.364595 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:34.364542    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:34.365075 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:34.364626    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:34.369174 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:34.369150    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:35.251914 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:35.251887    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:16:35.317891 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:35.317863    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85579c4694-sdcjp"]
Sep 18 19:16:39.269277 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:39.269245    2559 generic.go:356] "Generic (PLEG): container finished" podID="fbe7c860-1b27-480f-9de5-ba6eddfb76a0" containerID="c22aa5f8bbff5563e76e92f646b2ad1c9e9b3a529b8f450ed5a83e5a67332627" exitCode=0
Sep 18 19:16:39.269277 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:39.269288    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp" event={"ID":"fbe7c860-1b27-480f-9de5-ba6eddfb76a0","Type":"ContainerDied","Data":"c22aa5f8bbff5563e76e92f646b2ad1c9e9b3a529b8f450ed5a83e5a67332627"}
Sep 18 19:16:39.269721 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:39.269617    2559 scope.go:117] "RemoveContainer" containerID="c22aa5f8bbff5563e76e92f646b2ad1c9e9b3a529b8f450ed5a83e5a67332627"
Sep 18 19:16:40.274695 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:40.274659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-92gfp" event={"ID":"fbe7c860-1b27-480f-9de5-ba6eddfb76a0","Type":"ContainerStarted","Data":"d9aaa27a03a4248282979b85090aa16946a5a3c33cb1a13da01553f67d0fc481"}
Sep 18 19:16:49.311457 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:49.311423    2559 generic.go:356] "Generic (PLEG): container finished" podID="b6899294-bd1e-4b9c-ab8a-f10dfaba7a94" containerID="f1da04ec05ba8db9a027d10aea7e4e0c7f4282b543a23295cc72d5f322a25ec7" exitCode=0
Sep 18 19:16:49.311862 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:49.311498    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-kfskl" event={"ID":"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94","Type":"ContainerDied","Data":"f1da04ec05ba8db9a027d10aea7e4e0c7f4282b543a23295cc72d5f322a25ec7"}
Sep 18 19:16:49.311903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:49.311865    2559 scope.go:117] "RemoveContainer" containerID="f1da04ec05ba8db9a027d10aea7e4e0c7f4282b543a23295cc72d5f322a25ec7"
Sep 18 19:16:50.317337 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:16:50.317300    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-kfskl" event={"ID":"b6899294-bd1e-4b9c-ab8a-f10dfaba7a94","Type":"ContainerStarted","Data":"4c0402769edca6555b37638e360b7bfe3d84ba66c4cea5a618659e79ec1b2116"}
Sep 18 19:17:00.337487 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.337426    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85579c4694-sdcjp" podUID="379405c9-7e19-4eeb-8eaa-8f547b20d410" containerName="console" containerID="cri-o://23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75" gracePeriod=15
Sep 18 19:17:00.609784 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.609732    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:17:00.731699 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.731660    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-config\") pod \"379405c9-7e19-4eeb-8eaa-8f547b20d410\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") "
Sep 18 19:17:00.731879 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.731722    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-oauth-serving-cert\") pod \"379405c9-7e19-4eeb-8eaa-8f547b20d410\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") "
Sep 18 19:17:00.731879 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.731761    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-serving-cert\") pod \"379405c9-7e19-4eeb-8eaa-8f547b20d410\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") "
Sep 18 19:17:00.731879 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.731802    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-oauth-config\") pod \"379405c9-7e19-4eeb-8eaa-8f547b20d410\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") "
Sep 18 19:17:00.731879 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.731858    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-trusted-ca-bundle\") pod \"379405c9-7e19-4eeb-8eaa-8f547b20d410\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") "
Sep 18 19:17:00.732387 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.731890    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wczbr\" (UniqueName: \"kubernetes.io/projected/379405c9-7e19-4eeb-8eaa-8f547b20d410-kube-api-access-wczbr\") pod \"379405c9-7e19-4eeb-8eaa-8f547b20d410\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") "
Sep 18 19:17:00.732387 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.731921    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-service-ca\") pod \"379405c9-7e19-4eeb-8eaa-8f547b20d410\" (UID: \"379405c9-7e19-4eeb-8eaa-8f547b20d410\") "
Sep 18 19:17:00.732387 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.732005    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-config" (OuterVolumeSpecName: "console-config") pod "379405c9-7e19-4eeb-8eaa-8f547b20d410" (UID: "379405c9-7e19-4eeb-8eaa-8f547b20d410"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:00.732387 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.732109    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "379405c9-7e19-4eeb-8eaa-8f547b20d410" (UID: "379405c9-7e19-4eeb-8eaa-8f547b20d410"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:00.732387 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.732223    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:17:00.732387 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.732242    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-oauth-serving-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:17:00.732633 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.732503    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "379405c9-7e19-4eeb-8eaa-8f547b20d410" (UID: "379405c9-7e19-4eeb-8eaa-8f547b20d410"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:00.732633 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.732561    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-service-ca" (OuterVolumeSpecName: "service-ca") pod "379405c9-7e19-4eeb-8eaa-8f547b20d410" (UID: "379405c9-7e19-4eeb-8eaa-8f547b20d410"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:00.734627 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.734572    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "379405c9-7e19-4eeb-8eaa-8f547b20d410" (UID: "379405c9-7e19-4eeb-8eaa-8f547b20d410"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:00.734730 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.734673    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379405c9-7e19-4eeb-8eaa-8f547b20d410-kube-api-access-wczbr" (OuterVolumeSpecName: "kube-api-access-wczbr") pod "379405c9-7e19-4eeb-8eaa-8f547b20d410" (UID: "379405c9-7e19-4eeb-8eaa-8f547b20d410"). InnerVolumeSpecName "kube-api-access-wczbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:00.734730 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.734691    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "379405c9-7e19-4eeb-8eaa-8f547b20d410" (UID: "379405c9-7e19-4eeb-8eaa-8f547b20d410"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:00.832926 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.832896    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-trusted-ca-bundle\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:17:00.832926 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.832922    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wczbr\" (UniqueName: \"kubernetes.io/projected/379405c9-7e19-4eeb-8eaa-8f547b20d410-kube-api-access-wczbr\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:17:00.832926 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.832932    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/379405c9-7e19-4eeb-8eaa-8f547b20d410-service-ca\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:17:00.833153 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.832940    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-serving-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:17:00.833153 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:00.832949    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/379405c9-7e19-4eeb-8eaa-8f547b20d410-console-oauth-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:17:01.364639 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:01.364599    2559 generic.go:356] "Generic (PLEG): container finished" podID="379405c9-7e19-4eeb-8eaa-8f547b20d410" containerID="23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75" exitCode=0
Sep 18 19:17:01.365036 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:01.364681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85579c4694-sdcjp" event={"ID":"379405c9-7e19-4eeb-8eaa-8f547b20d410","Type":"ContainerDied","Data":"23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75"}
Sep 18 19:17:01.365036 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:01.364696    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85579c4694-sdcjp"
Sep 18 19:17:01.365036 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:01.364751    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85579c4694-sdcjp" event={"ID":"379405c9-7e19-4eeb-8eaa-8f547b20d410","Type":"ContainerDied","Data":"ee73d110ebcec29b27fe26b22e530358c4cc140d54794f5f2d45271a4871b2f0"}
Sep 18 19:17:01.365036 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:01.364786    2559 scope.go:117] "RemoveContainer" containerID="23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75"
Sep 18 19:17:01.375009 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:01.374992    2559 scope.go:117] "RemoveContainer" containerID="23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75"
Sep 18 19:17:01.375330 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:17:01.375307    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75\": container with ID starting with 23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75 not found: ID does not exist" containerID="23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75"
Sep 18 19:17:01.375425 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:01.375340    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75"} err="failed to get container status \"23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75\": rpc error: code = NotFound desc = could not find container \"23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75\": container with ID starting with 23b7109765b497d4e2211b75b9b3bd6bf4850c442f2085f84b1a71f45279ad75 not found: ID does not exist"
Sep 18 19:17:01.402881 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:01.402851    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85579c4694-sdcjp"]
Sep 18 19:17:01.412838 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:01.412810    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85579c4694-sdcjp"]
Sep 18 19:17:01.711401 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:17:01.711316    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="379405c9-7e19-4eeb-8eaa-8f547b20d410" path="/var/lib/kubelet/pods/379405c9-7e19-4eeb-8eaa-8f547b20d410/volumes"
Sep 18 19:19:39.593763 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:19:39.593734    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:19:39.598303 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:19:39.598276    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:19:39.605462 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:19:39.605441    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:21:24.903695 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:24.903663    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:21:24.904222 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:24.903931    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="alertmanager" containerID="cri-o://287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20" gracePeriod=120
Sep 18 19:21:24.904222 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:24.904000    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy-metric" containerID="cri-o://ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806" gracePeriod=120
Sep 18 19:21:24.904222 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:24.904080    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="config-reloader" containerID="cri-o://5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d" gracePeriod=120
Sep 18 19:21:24.904222 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:24.904000    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy-web" containerID="cri-o://d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b" gracePeriod=120
Sep 18 19:21:24.904222 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:24.904102    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy" containerID="cri-o://bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a" gracePeriod=120
Sep 18 19:21:24.904222 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:24.904181    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="prom-label-proxy" containerID="cri-o://f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e" gracePeriod=120
Sep 18 19:21:25.151414 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.151386    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.276251 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276229    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-main-tls\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276445 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276263    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8597v\" (UniqueName: \"kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-kube-api-access-8597v\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276445 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276289    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-out\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276445 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276318    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-volume\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276445 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-metrics-client-ca\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276445 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276402    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-web\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276692 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276455    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-trusted-ca-bundle\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276692 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276487    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-web-config\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276692 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-cluster-tls-config\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276692 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276543    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-main-db\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276692 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276568    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276692 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276600    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-tls-assets\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.276692 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276636    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy\") pod \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\" (UID: \"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4\") "
Sep 18 19:21:25.277039 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276748    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:21:25.277039 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.276926    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-metrics-client-ca\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.277571 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.277536    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:21:25.277884 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.277850    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:21:25.279152 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.279125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-out" (OuterVolumeSpecName: "config-out") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:21:25.279825 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.279715    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:25.279825 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.279761    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:25.279825 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.279799    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-volume" (OuterVolumeSpecName: "config-volume") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:25.280025 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.279821    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-kube-api-access-8597v" (OuterVolumeSpecName: "kube-api-access-8597v") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "kube-api-access-8597v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:21:25.280402 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.280374    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:25.280474 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.280403    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:25.280584 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.280567    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:21:25.283704 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.283684    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:25.291458 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.291438    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-web-config" (OuterVolumeSpecName: "web-config") pod "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" (UID: "e435197f-98bb-4df0-8fc6-8cba4f3bf5e4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:25.365553 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365524    2559 generic.go:356] "Generic (PLEG): container finished" podID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerID="f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e" exitCode=0
Sep 18 19:21:25.365553 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365546    2559 generic.go:356] "Generic (PLEG): container finished" podID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerID="ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806" exitCode=0
Sep 18 19:21:25.365553 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365553    2559 generic.go:356] "Generic (PLEG): container finished" podID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerID="bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a" exitCode=0
Sep 18 19:21:25.365553 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365558    2559 generic.go:356] "Generic (PLEG): container finished" podID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerID="d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b" exitCode=0
Sep 18 19:21:25.365759 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365563    2559 generic.go:356] "Generic (PLEG): container finished" podID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerID="5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d" exitCode=0
Sep 18 19:21:25.365759 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365569    2559 generic.go:356] "Generic (PLEG): container finished" podID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerID="287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20" exitCode=0
Sep 18 19:21:25.365759 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365610    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerDied","Data":"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"}
Sep 18 19:21:25.365759 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365673    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerDied","Data":"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"}
Sep 18 19:21:25.365759 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365616    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.365759 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365702    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerDied","Data":"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"}
Sep 18 19:21:25.365759 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerDied","Data":"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"}
Sep 18 19:21:25.365759 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerDied","Data":"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"}
Sep 18 19:21:25.366074 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365772    2559 scope.go:117] "RemoveContainer" containerID="f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"
Sep 18 19:21:25.366074 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365778    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerDied","Data":"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"}
Sep 18 19:21:25.366074 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.365884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e435197f-98bb-4df0-8fc6-8cba4f3bf5e4","Type":"ContainerDied","Data":"b1f7e224f767ef7f2fe7dc7d0f210852ca72bd20a5b4f5320cf11d1b7f6091e7"}
Sep 18 19:21:25.373257 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.373237    2559 scope.go:117] "RemoveContainer" containerID="ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"
Sep 18 19:21:25.377821 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377799    2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377821 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377818    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-web-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377828    2559 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-cluster-tls-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377836    2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-alertmanager-main-db\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377845    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377854    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-tls-assets\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377862    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377870    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-main-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377878    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8597v\" (UniqueName: \"kubernetes.io/projected/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-kube-api-access-8597v\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377886    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-out\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377894    2559 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-config-volume\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.377962 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.377902    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:21:25.380339 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.380323    2559 scope.go:117] "RemoveContainer" containerID="bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"
Sep 18 19:21:25.386901 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.386882    2559 scope.go:117] "RemoveContainer" containerID="d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"
Sep 18 19:21:25.392979 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.392963    2559 scope.go:117] "RemoveContainer" containerID="5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"
Sep 18 19:21:25.397691 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.397665    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:21:25.400031 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.400015    2559 scope.go:117] "RemoveContainer" containerID="287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"
Sep 18 19:21:25.404450 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.404432    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:21:25.408967 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.408953    2559 scope.go:117] "RemoveContainer" containerID="5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"
Sep 18 19:21:25.416258 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.416237    2559 scope.go:117] "RemoveContainer" containerID="f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"
Sep 18 19:21:25.416527 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:21:25.416511    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": container with ID starting with f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e not found: ID does not exist" containerID="f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"
Sep 18 19:21:25.416581 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.416536    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"} err="failed to get container status \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": rpc error: code = NotFound desc = could not find container \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": container with ID starting with f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e not found: ID does not exist"
Sep 18 19:21:25.416581 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.416552    2559 scope.go:117] "RemoveContainer" containerID="ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"
Sep 18 19:21:25.416785 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:21:25.416767    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": container with ID starting with ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806 not found: ID does not exist" containerID="ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"
Sep 18 19:21:25.416834 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.416791    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"} err="failed to get container status \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": rpc error: code = NotFound desc = could not find container \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": container with ID starting with ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806 not found: ID does not exist"
Sep 18 19:21:25.416834 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.416810    2559 scope.go:117] "RemoveContainer" containerID="bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"
Sep 18 19:21:25.417060 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:21:25.417043    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": container with ID starting with bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a not found: ID does not exist" containerID="bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"
Sep 18 19:21:25.417095 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.417064    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"} err="failed to get container status \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": rpc error: code = NotFound desc = could not find container \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": container with ID starting with bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a not found: ID does not exist"
Sep 18 19:21:25.417095 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.417078    2559 scope.go:117] "RemoveContainer" containerID="d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"
Sep 18 19:21:25.417311 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:21:25.417295    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": container with ID starting with d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b not found: ID does not exist" containerID="d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"
Sep 18 19:21:25.417381 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.417313    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"} err="failed to get container status \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": rpc error: code = NotFound desc = could not find container \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": container with ID starting with d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b not found: ID does not exist"
Sep 18 19:21:25.417381 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.417325    2559 scope.go:117] "RemoveContainer" containerID="5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"
Sep 18 19:21:25.417537 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:21:25.417517    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": container with ID starting with 5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d not found: ID does not exist" containerID="5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"
Sep 18 19:21:25.417628 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.417539    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"} err="failed to get container status \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": rpc error: code = NotFound desc = could not find container \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": container with ID starting with 5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d not found: ID does not exist"
Sep 18 19:21:25.417628 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.417556    2559 scope.go:117] "RemoveContainer" containerID="287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"
Sep 18 19:21:25.417773 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:21:25.417757    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": container with ID starting with 287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20 not found: ID does not exist" containerID="287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"
Sep 18 19:21:25.417818 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.417776    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"} err="failed to get container status \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": rpc error: code = NotFound desc = could not find container \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": container with ID starting with 287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20 not found: ID does not exist"
Sep 18 19:21:25.417818 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.417798    2559 scope.go:117] "RemoveContainer" containerID="5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"
Sep 18 19:21:25.418033 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:21:25.418018    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": container with ID starting with 5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51 not found: ID does not exist" containerID="5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"
Sep 18 19:21:25.418075 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.418038    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"} err="failed to get container status \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": rpc error: code = NotFound desc = could not find container \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": container with ID starting with 5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51 not found: ID does not exist"
Sep 18 19:21:25.418075 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.418052    2559 scope.go:117] "RemoveContainer" containerID="f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"
Sep 18 19:21:25.418284 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.418267    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"} err="failed to get container status \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": rpc error: code = NotFound desc = could not find container \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": container with ID starting with f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e not found: ID does not exist"
Sep 18 19:21:25.418332 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.418284    2559 scope.go:117] "RemoveContainer" containerID="ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"
Sep 18 19:21:25.418504 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.418486    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"} err="failed to get container status \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": rpc error: code = NotFound desc = could not find container \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": container with ID starting with ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806 not found: ID does not exist"
Sep 18 19:21:25.418563 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.418504    2559 scope.go:117] "RemoveContainer" containerID="bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"
Sep 18 19:21:25.418761 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.418742    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"} err="failed to get container status \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": rpc error: code = NotFound desc = could not find container \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": container with ID starting with bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a not found: ID does not exist"
Sep 18 19:21:25.418806 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.418763    2559 scope.go:117] "RemoveContainer" containerID="d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"
Sep 18 19:21:25.418981 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.418966    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"} err="failed to get container status \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": rpc error: code = NotFound desc = could not find container \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": container with ID starting with d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b not found: ID does not exist"
Sep 18 19:21:25.418981 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.418981    2559 scope.go:117] "RemoveContainer" containerID="5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"
Sep 18 19:21:25.419260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.419236    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"} err="failed to get container status \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": rpc error: code = NotFound desc = could not find container \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": container with ID starting with 5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d not found: ID does not exist"
Sep 18 19:21:25.419334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.419262    2559 scope.go:117] "RemoveContainer" containerID="287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"
Sep 18 19:21:25.419502 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.419478    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"} err="failed to get container status \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": rpc error: code = NotFound desc = could not find container \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": container with ID starting with 287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20 not found: ID does not exist"
Sep 18 19:21:25.419502 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.419497    2559 scope.go:117] "RemoveContainer" containerID="5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"
Sep 18 19:21:25.419746 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.419731    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"} err="failed to get container status \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": rpc error: code = NotFound desc = could not find container \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": container with ID starting with 5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51 not found: ID does not exist"
Sep 18 19:21:25.419796 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.419747    2559 scope.go:117] "RemoveContainer" containerID="f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"
Sep 18 19:21:25.419949 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.419930    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"} err="failed to get container status \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": rpc error: code = NotFound desc = could not find container \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": container with ID starting with f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e not found: ID does not exist"
Sep 18 19:21:25.419993 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.419948    2559 scope.go:117] "RemoveContainer" containerID="ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"
Sep 18 19:21:25.420169 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.420154    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"} err="failed to get container status \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": rpc error: code = NotFound desc = could not find container \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": container with ID starting with ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806 not found: ID does not exist"
Sep 18 19:21:25.420229 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.420170    2559 scope.go:117] "RemoveContainer" containerID="bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"
Sep 18 19:21:25.420426 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.420398    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"} err="failed to get container status \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": rpc error: code = NotFound desc = could not find container \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": container with ID starting with bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a not found: ID does not exist"
Sep 18 19:21:25.420426 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.420426    2559 scope.go:117] "RemoveContainer" containerID="d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"
Sep 18 19:21:25.420654 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.420639    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"} err="failed to get container status \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": rpc error: code = NotFound desc = could not find container \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": container with ID starting with d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b not found: ID does not exist"
Sep 18 19:21:25.420707 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.420654    2559 scope.go:117] "RemoveContainer" containerID="5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"
Sep 18 19:21:25.420854 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.420838    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"} err="failed to get container status \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": rpc error: code = NotFound desc = could not find container \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": container with ID starting with 5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d not found: ID does not exist"
Sep 18 19:21:25.420895 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.420853    2559 scope.go:117] "RemoveContainer" containerID="287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"
Sep 18 19:21:25.421076 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.421054    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"} err="failed to get container status \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": rpc error: code = NotFound desc = could not find container \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": container with ID starting with 287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20 not found: ID does not exist"
Sep 18 19:21:25.421076 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.421074    2559 scope.go:117] "RemoveContainer" containerID="5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"
Sep 18 19:21:25.421311 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.421293    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"} err="failed to get container status \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": rpc error: code = NotFound desc = could not find container \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": container with ID starting with 5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51 not found: ID does not exist"
Sep 18 19:21:25.421369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.421313    2559 scope.go:117] "RemoveContainer" containerID="f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"
Sep 18 19:21:25.421582 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.421564    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"} err="failed to get container status \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": rpc error: code = NotFound desc = could not find container \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": container with ID starting with f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e not found: ID does not exist"
Sep 18 19:21:25.421623 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.421583    2559 scope.go:117] "RemoveContainer" containerID="ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"
Sep 18 19:21:25.421792 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.421777    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"} err="failed to get container status \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": rpc error: code = NotFound desc = could not find container \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": container with ID starting with ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806 not found: ID does not exist"
Sep 18 19:21:25.421845 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.421794    2559 scope.go:117] "RemoveContainer" containerID="bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"
Sep 18 19:21:25.421997 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.421984    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"} err="failed to get container status \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": rpc error: code = NotFound desc = could not find container \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": container with ID starting with bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a not found: ID does not exist"
Sep 18 19:21:25.422046 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.421998    2559 scope.go:117] "RemoveContainer" containerID="d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"
Sep 18 19:21:25.422237 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.422214    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"} err="failed to get container status \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": rpc error: code = NotFound desc = could not find container \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": container with ID starting with d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b not found: ID does not exist"
Sep 18 19:21:25.422302 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.422237    2559 scope.go:117] "RemoveContainer" containerID="5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"
Sep 18 19:21:25.422483 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.422464    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"} err="failed to get container status \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": rpc error: code = NotFound desc = could not find container \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": container with ID starting with 5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d not found: ID does not exist"
Sep 18 19:21:25.422483 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.422482    2559 scope.go:117] "RemoveContainer" containerID="287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"
Sep 18 19:21:25.422713 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.422698    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"} err="failed to get container status \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": rpc error: code = NotFound desc = could not find container \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": container with ID starting with 287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20 not found: ID does not exist"
Sep 18 19:21:25.422761 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.422714    2559 scope.go:117] "RemoveContainer" containerID="5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"
Sep 18 19:21:25.422919 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.422901    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"} err="failed to get container status \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": rpc error: code = NotFound desc = could not find container \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": container with ID starting with 5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51 not found: ID does not exist"
Sep 18 19:21:25.422919 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.422919    2559 scope.go:117] "RemoveContainer" containerID="f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"
Sep 18 19:21:25.423147 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.423126    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"} err="failed to get container status \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": rpc error: code = NotFound desc = could not find container \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": container with ID starting with f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e not found: ID does not exist"
Sep 18 19:21:25.423147 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.423141    2559 scope.go:117] "RemoveContainer" containerID="ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"
Sep 18 19:21:25.423337 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.423320    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"} err="failed to get container status \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": rpc error: code = NotFound desc = could not find container \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": container with ID starting with ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806 not found: ID does not exist"
Sep 18 19:21:25.423382 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.423336    2559 scope.go:117] "RemoveContainer" containerID="bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"
Sep 18 19:21:25.423537 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.423517    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"} err="failed to get container status \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": rpc error: code = NotFound desc = could not find container \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": container with ID starting with bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a not found: ID does not exist"
Sep 18 19:21:25.423578 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.423537    2559 scope.go:117] "RemoveContainer" containerID="d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"
Sep 18 19:21:25.423727 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.423711    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"} err="failed to get container status \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": rpc error: code = NotFound desc = could not find container \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": container with ID starting with d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b not found: ID does not exist"
Sep 18 19:21:25.423768 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.423728    2559 scope.go:117] "RemoveContainer" containerID="5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"
Sep 18 19:21:25.423976 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.423948    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"} err="failed to get container status \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": rpc error: code = NotFound desc = could not find container \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": container with ID starting with 5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d not found: ID does not exist"
Sep 18 19:21:25.423976 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.423976    2559 scope.go:117] "RemoveContainer" containerID="287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"
Sep 18 19:21:25.424187 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.424171    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"} err="failed to get container status \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": rpc error: code = NotFound desc = could not find container \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": container with ID starting with 287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20 not found: ID does not exist"
Sep 18 19:21:25.424256 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.424188    2559 scope.go:117] "RemoveContainer" containerID="5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"
Sep 18 19:21:25.424406 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.424390    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"} err="failed to get container status \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": rpc error: code = NotFound desc = could not find container \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": container with ID starting with 5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51 not found: ID does not exist"
Sep 18 19:21:25.424406 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.424404    2559 scope.go:117] "RemoveContainer" containerID="f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"
Sep 18 19:21:25.424600 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.424586    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e"} err="failed to get container status \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": rpc error: code = NotFound desc = could not find container \"f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e\": container with ID starting with f6eaccf3a20abf0c6cb088d9739b17adcf1578814866162b0e0d0eb3ed5f9d8e not found: ID does not exist"
Sep 18 19:21:25.424642 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.424600    2559 scope.go:117] "RemoveContainer" containerID="ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"
Sep 18 19:21:25.424785 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.424770    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806"} err="failed to get container status \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": rpc error: code = NotFound desc = could not find container \"ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806\": container with ID starting with ca4d94b9dba601660cb3ca60dec52c77712a41b075316800781bea12bbef6806 not found: ID does not exist"
Sep 18 19:21:25.424831 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.424784    2559 scope.go:117] "RemoveContainer" containerID="bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"
Sep 18 19:21:25.425012 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.424995    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a"} err="failed to get container status \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": rpc error: code = NotFound desc = could not find container \"bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a\": container with ID starting with bd5f4af7a14a42297a2989a974fd33c1304bf422704a8ba1ff1e2d0bb064707a not found: ID does not exist"
Sep 18 19:21:25.425104 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.425011    2559 scope.go:117] "RemoveContainer" containerID="d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"
Sep 18 19:21:25.425239 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.425225    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b"} err="failed to get container status \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": rpc error: code = NotFound desc = could not find container \"d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b\": container with ID starting with d887fb0cfcf1fd3a016e8efc33af014629476c74bda4206e3024cee224f0643b not found: ID does not exist"
Sep 18 19:21:25.425298 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.425239    2559 scope.go:117] "RemoveContainer" containerID="5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"
Sep 18 19:21:25.425473 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.425453    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d"} err="failed to get container status \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": rpc error: code = NotFound desc = could not find container \"5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d\": container with ID starting with 5598ae7fbca569a9b0692f13240f309b93ce7d1a23682c8c2517ce5795197b7d not found: ID does not exist"
Sep 18 19:21:25.425518 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.425474    2559 scope.go:117] "RemoveContainer" containerID="287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"
Sep 18 19:21:25.425683 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.425667    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20"} err="failed to get container status \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": rpc error: code = NotFound desc = could not find container \"287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20\": container with ID starting with 287f72622010ff77d916c65d10b7c4ebfec4efecb374fbb7b65f016ac8c92b20 not found: ID does not exist"
Sep 18 19:21:25.425683 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.425682    2559 scope.go:117] "RemoveContainer" containerID="5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"
Sep 18 19:21:25.425889 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.425869    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51"} err="failed to get container status \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": rpc error: code = NotFound desc = could not find container \"5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51\": container with ID starting with 5f8d092da2922e4856d0e7abb85b5717caf52d3e3e6a5bcf1ebcf41ef7795a51 not found: ID does not exist"
Sep 18 19:21:25.467195 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467164    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:21:25.467519 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467502    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379405c9-7e19-4eeb-8eaa-8f547b20d410" containerName="console"
Sep 18 19:21:25.467594 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467521    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="379405c9-7e19-4eeb-8eaa-8f547b20d410" containerName="console"
Sep 18 19:21:25.467594 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467533    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy-web"
Sep 18 19:21:25.467594 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467541    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy-web"
Sep 18 19:21:25.467594 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467555    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="config-reloader"
Sep 18 19:21:25.467594 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467564    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="config-reloader"
Sep 18 19:21:25.467594 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467579    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="prom-label-proxy"
Sep 18 19:21:25.467594 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467587    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="prom-label-proxy"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467600    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="init-config-reloader"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467608    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="init-config-reloader"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467619    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy-metric"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467628    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy-metric"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467640    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="alertmanager"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467649    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="alertmanager"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467657    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467665    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467734    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="alertmanager"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467747    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy-metric"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467758    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="379405c9-7e19-4eeb-8eaa-8f547b20d410" containerName="console"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467766    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="config-reloader"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467775    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy-web"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467785    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="kube-rbac-proxy"
Sep 18 19:21:25.467912 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.467794    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" containerName="prom-label-proxy"
Sep 18 19:21:25.471752 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.471732    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.476363 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.476346    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 19:21:25.476491 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.476475    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 19:21:25.479215 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.479182    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 19:21:25.479337 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.479322    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 19:21:25.479966 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.479949    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 19:21:25.480326 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.480309    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 19:21:25.481271 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.481257    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-626rt\""
Sep 18 19:21:25.481484 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.481466    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 19:21:25.481745 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.481731    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 19:21:25.483863 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.483846    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 19:21:25.517415 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.517384    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:21:25.578925 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.578832    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.578925 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.578882    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ca01eda-ec1d-4f62-a1ff-e59798842c65-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.578932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.578958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxswg\" (UniqueName: \"kubernetes.io/projected/0ca01eda-ec1d-4f62-a1ff-e59798842c65-kube-api-access-hxswg\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.579040    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.579081    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ca01eda-ec1d-4f62-a1ff-e59798842c65-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.579100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.579115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ca01eda-ec1d-4f62-a1ff-e59798842c65-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579371 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.579156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ca01eda-ec1d-4f62-a1ff-e59798842c65-config-out\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579371 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.579191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-web-config\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579371 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.579255    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579371 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.579279    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.579371 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.579327    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ca01eda-ec1d-4f62-a1ff-e59798842c65-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.680425 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.680425 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ca01eda-ec1d-4f62-a1ff-e59798842c65-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.680634 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.680634 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680458    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxswg\" (UniqueName: \"kubernetes.io/projected/0ca01eda-ec1d-4f62-a1ff-e59798842c65-kube-api-access-hxswg\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.680634 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.680634 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ca01eda-ec1d-4f62-a1ff-e59798842c65-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.680634 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680558    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.680634 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680578    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ca01eda-ec1d-4f62-a1ff-e59798842c65-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.680634 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680608    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ca01eda-ec1d-4f62-a1ff-e59798842c65-config-out\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.680634 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-web-config\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.681063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.681063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.681063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ca01eda-ec1d-4f62-a1ff-e59798842c65-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.681063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.680856    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ca01eda-ec1d-4f62-a1ff-e59798842c65-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.681459 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.681425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ca01eda-ec1d-4f62-a1ff-e59798842c65-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.682003 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.681619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ca01eda-ec1d-4f62-a1ff-e59798842c65-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.683968 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.683686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.683968 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.683867    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.683968 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.683939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-web-config\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.684173 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.683981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.684173 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.684009    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.684311 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.684235    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.684367 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.684330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ca01eda-ec1d-4f62-a1ff-e59798842c65-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.684981 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.684954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ca01eda-ec1d-4f62-a1ff-e59798842c65-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.685963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.685943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ca01eda-ec1d-4f62-a1ff-e59798842c65-config-out\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.691121 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.691105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxswg\" (UniqueName: \"kubernetes.io/projected/0ca01eda-ec1d-4f62-a1ff-e59798842c65-kube-api-access-hxswg\") pod \"alertmanager-main-0\" (UID: \"0ca01eda-ec1d-4f62-a1ff-e59798842c65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:25.713676 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.710774    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e435197f-98bb-4df0-8fc6-8cba4f3bf5e4" path="/var/lib/kubelet/pods/e435197f-98bb-4df0-8fc6-8cba4f3bf5e4/volumes"
Sep 18 19:21:25.781297 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:25.781265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:21:26.071287 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:26.071259    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:21:26.072907 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:21:26.072882    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca01eda_ec1d_4f62_a1ff_e59798842c65.slice/crio-fe2d3ee79405cd4bad41c8cc360e8d3786a414e68b33ef4f0ea2772951bdae2d WatchSource:0}: Error finding container fe2d3ee79405cd4bad41c8cc360e8d3786a414e68b33ef4f0ea2772951bdae2d: Status 404 returned error can't find the container with id fe2d3ee79405cd4bad41c8cc360e8d3786a414e68b33ef4f0ea2772951bdae2d
Sep 18 19:21:26.370766 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:26.370737    2559 generic.go:356] "Generic (PLEG): container finished" podID="0ca01eda-ec1d-4f62-a1ff-e59798842c65" containerID="d980f3f69c9580ee7c8d32a48b83a4419fe273630a336283973479c4ea679d97" exitCode=0
Sep 18 19:21:26.370940 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:26.370834    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ca01eda-ec1d-4f62-a1ff-e59798842c65","Type":"ContainerDied","Data":"d980f3f69c9580ee7c8d32a48b83a4419fe273630a336283973479c4ea679d97"}
Sep 18 19:21:26.370940 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:26.370907    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ca01eda-ec1d-4f62-a1ff-e59798842c65","Type":"ContainerStarted","Data":"fe2d3ee79405cd4bad41c8cc360e8d3786a414e68b33ef4f0ea2772951bdae2d"}
Sep 18 19:21:27.380214 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:27.380157    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ca01eda-ec1d-4f62-a1ff-e59798842c65","Type":"ContainerStarted","Data":"87dafac7082ce31db9c2cf0dcaa3bc1f5f7c4cfdbd5eb9d4194dbdd34aabc174"}
Sep 18 19:21:27.380590 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:27.380249    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ca01eda-ec1d-4f62-a1ff-e59798842c65","Type":"ContainerStarted","Data":"5fc0cdacfd5a32e5193dbc7a5d8c76ed6e695cccd698be8345c927b177eea2fa"}
Sep 18 19:21:27.380590 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:27.380284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ca01eda-ec1d-4f62-a1ff-e59798842c65","Type":"ContainerStarted","Data":"2109c88d7375486b309d5a10ae3b242647c78e3a7301c025ecc69f11c5d4be04"}
Sep 18 19:21:27.380590 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:27.380310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ca01eda-ec1d-4f62-a1ff-e59798842c65","Type":"ContainerStarted","Data":"9ca2f38f07363f2581eff8eaaa11b19005252623527f427ff2e91dd2bebea3ce"}
Sep 18 19:21:27.380590 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:27.380332    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ca01eda-ec1d-4f62-a1ff-e59798842c65","Type":"ContainerStarted","Data":"f187ce7ce2ac1cc47a2092c26150c65549e6e6f454ddb107e30f7c8de38db8e9"}
Sep 18 19:21:27.380590 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:27.380355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ca01eda-ec1d-4f62-a1ff-e59798842c65","Type":"ContainerStarted","Data":"84f832c03982dcc9b06805474699a8e68aff0d8d25e2a4a94f5b417728e4cbae"}
Sep 18 19:21:27.432908 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:27.432852    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.415386801 podStartE2EDuration="2.432835178s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.448377ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:21:27.429003646 +0000 UTC m=+409.883141837" watchObservedRunningTime="2026-09-18 19:21:27.432835178 +0000 UTC m=+409.886973355"
Sep 18 19:21:28.988537 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:28.988503    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"]
Sep 18 19:21:28.995927 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:28.995899    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.012059 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.012038    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 19:21:29.012247 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.012132    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 19:21:29.012247 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.012175    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 19:21:29.012426 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.012410    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 19:21:29.012683 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.012670    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-q8hpw\""
Sep 18 19:21:29.012918 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.012906    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 19:21:29.028367 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.028349    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 19:21:29.046909 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.046885    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"]
Sep 18 19:21:29.108489 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.108466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.108622 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.108498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-secret-telemeter-client\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.108622 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.108514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-federate-client-tls\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.108622 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.108529    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6cb263b-7557-4f6f-8601-f31c0b16fe54-serving-certs-ca-bundle\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.108622 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.108582    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hc8p\" (UniqueName: \"kubernetes.io/projected/a6cb263b-7557-4f6f-8601-f31c0b16fe54-kube-api-access-9hc8p\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.108622 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.108617    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-telemeter-client-tls\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.108836 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.108633    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6cb263b-7557-4f6f-8601-f31c0b16fe54-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.108836 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.108722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6cb263b-7557-4f6f-8601-f31c0b16fe54-metrics-client-ca\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.209812 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.209766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6cb263b-7557-4f6f-8601-f31c0b16fe54-metrics-client-ca\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.209812 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.209816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.210064 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.209838    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-secret-telemeter-client\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.210064 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.209854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-federate-client-tls\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.210064 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.209870    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6cb263b-7557-4f6f-8601-f31c0b16fe54-serving-certs-ca-bundle\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.210064 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.209902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hc8p\" (UniqueName: \"kubernetes.io/projected/a6cb263b-7557-4f6f-8601-f31c0b16fe54-kube-api-access-9hc8p\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.210064 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.209954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-telemeter-client-tls\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.210064 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.209982    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6cb263b-7557-4f6f-8601-f31c0b16fe54-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.210831 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.210796    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6cb263b-7557-4f6f-8601-f31c0b16fe54-metrics-client-ca\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.210946 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.210910    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6cb263b-7557-4f6f-8601-f31c0b16fe54-serving-certs-ca-bundle\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.211467 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.211441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6cb263b-7557-4f6f-8601-f31c0b16fe54-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.212624 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.212597    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.212706 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.212623    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-secret-telemeter-client\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.212784 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.212763    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-telemeter-client-tls\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.212823 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.212763    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a6cb263b-7557-4f6f-8601-f31c0b16fe54-federate-client-tls\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.224808 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.224790    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hc8p\" (UniqueName: \"kubernetes.io/projected/a6cb263b-7557-4f6f-8601-f31c0b16fe54-kube-api-access-9hc8p\") pod \"telemeter-client-fcbf478c9-gtx5k\" (UID: \"a6cb263b-7557-4f6f-8601-f31c0b16fe54\") " pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.305502 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.305474    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"
Sep 18 19:21:29.604815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.604784    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fcbf478c9-gtx5k"]
Sep 18 19:21:29.605157 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:21:29.605137    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cb263b_7557_4f6f_8601_f31c0b16fe54.slice/crio-d92b00520adabb4a0c0950b6f0e4db986dd2f61e860033ac4dbc74e7723a453d WatchSource:0}: Error finding container d92b00520adabb4a0c0950b6f0e4db986dd2f61e860033ac4dbc74e7723a453d: Status 404 returned error can't find the container with id d92b00520adabb4a0c0950b6f0e4db986dd2f61e860033ac4dbc74e7723a453d
Sep 18 19:21:29.607138 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:29.607121    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:21:30.393852 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.393812    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k" event={"ID":"a6cb263b-7557-4f6f-8601-f31c0b16fe54","Type":"ContainerStarted","Data":"d92b00520adabb4a0c0950b6f0e4db986dd2f61e860033ac4dbc74e7723a453d"}
Sep 18 19:21:30.534585 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.534550    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:21:30.537964 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.537939    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.573908 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.573876    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:21:30.573908 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.573899    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:21:30.581023 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.580995    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:21:30.581167 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.581127    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:21:30.581167 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.581162    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:21:30.581290 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.581271    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:21:30.588389 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.588367    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:21:30.588493 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.588399    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:21:30.591054 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.591014    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f9b4c\""
Sep 18 19:21:30.591263 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.591240    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ejrk81aje76u4\""
Sep 18 19:21:30.591356 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.591279    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:21:30.591356 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.591287    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:21:30.591468 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.591365    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:21:30.591526 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.591488    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:21:30.591526 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.591516    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:21:30.607082 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.607056    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:21:30.612909 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.612889    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:21:30.619864 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.619837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.619957 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.619874    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.619957 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.619911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-config\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.619957 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.619936    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620116 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.619969    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620116 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620001    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620116 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620027    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620116 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620116 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620079    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nzq9\" (UniqueName: \"kubernetes.io/projected/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-kube-api-access-2nzq9\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620116 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620111    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620413 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620182    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:21:30.620413 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620413 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620413 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620413 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620292    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620413 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620349    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620413 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620379    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620711 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620711 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620494    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620711 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.620711 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.620602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.721852 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.721781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.721852 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.721809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.721852 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.721832    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.721852 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.721849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.721870    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.721884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.721905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.721941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.721959    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.721988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-config\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722105    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722131    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722689 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722689 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722180    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722689 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722228    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722689 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nzq9\" (UniqueName: \"kubernetes.io/projected/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-kube-api-access-2nzq9\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722689 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722689 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722291    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.722689 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722620    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.723028 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722697    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.723028 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.722872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.725119 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.724770    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.725119 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.724904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.725333 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.725161    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-config\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.725530 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.725503    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.725530 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.725506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.725708 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.725579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.726105 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.726069    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.726589 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.726247    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.726726 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.726707    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.727141 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.727121    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.727141 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.727128    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.727588 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.727563    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.727752 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.727736    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.727791 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.727755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.728219 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.728184    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.819055 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.819021    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nzq9\" (UniqueName: \"kubernetes.io/projected/8bbbeb7c-05f6-45f1-9a03-9e74ac903bab-kube-api-access-2nzq9\") pod \"prometheus-k8s-0\" (UID: \"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:30.849136 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:30.849099    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:31.227504 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:31.227471    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:21:31.375964 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:21:31.375930    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbbeb7c_05f6_45f1_9a03_9e74ac903bab.slice/crio-7e65eb6a323af7fd1a3e6d6a48e8bfa051fd34bc4efc24c915a01728731d06f2 WatchSource:0}: Error finding container 7e65eb6a323af7fd1a3e6d6a48e8bfa051fd34bc4efc24c915a01728731d06f2: Status 404 returned error can't find the container with id 7e65eb6a323af7fd1a3e6d6a48e8bfa051fd34bc4efc24c915a01728731d06f2
Sep 18 19:21:31.404076 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:31.404049    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab","Type":"ContainerStarted","Data":"7e65eb6a323af7fd1a3e6d6a48e8bfa051fd34bc4efc24c915a01728731d06f2"}
Sep 18 19:21:32.410593 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:32.410554    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k" event={"ID":"a6cb263b-7557-4f6f-8601-f31c0b16fe54","Type":"ContainerStarted","Data":"b46d3ab5d37239dc90afca857d78fd7105c9df158bf21e099003f32ecbdd23b6"}
Sep 18 19:21:32.410982 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:32.410616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k" event={"ID":"a6cb263b-7557-4f6f-8601-f31c0b16fe54","Type":"ContainerStarted","Data":"80321ba23184bce3bbfa962e2d66d960927cd5215de20c93ea752d0e33898ec0"}
Sep 18 19:21:32.410982 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:32.410635    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k" event={"ID":"a6cb263b-7557-4f6f-8601-f31c0b16fe54","Type":"ContainerStarted","Data":"a96c40cd6c7f20db1e974f3bc6f1d2b7dc3003f0befaf9028e6d18b5a461408f"}
Sep 18 19:21:32.412163 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:32.412141    2559 generic.go:356] "Generic (PLEG): container finished" podID="8bbbeb7c-05f6-45f1-9a03-9e74ac903bab" containerID="d1cb787f1963cf55a1a55a8a00ae4dfd074228d812c9326fd3ae81490791d27b" exitCode=0
Sep 18 19:21:32.412240 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:32.412213    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab","Type":"ContainerDied","Data":"d1cb787f1963cf55a1a55a8a00ae4dfd074228d812c9326fd3ae81490791d27b"}
Sep 18 19:21:32.455642 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:32.455591    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-fcbf478c9-gtx5k" podStartSLOduration=2.654462429 podStartE2EDuration="4.455575526s" totalImagesPullingTime="1.801113097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:21:32.454459765 +0000 UTC m=+414.908597943" watchObservedRunningTime="2026-09-18 19:21:32.455575526 +0000 UTC m=+414.909713703"
Sep 18 19:21:35.432826 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:35.432786    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab","Type":"ContainerStarted","Data":"0854c3d5407bf863467c23ef5ffb6b546defb4ed603d2fc90b65dc84cdb95c89"}
Sep 18 19:21:35.433240 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:35.432935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab","Type":"ContainerStarted","Data":"2af59231153de43addf9b963de5191a62c1ff46abd6308a404735597363d7507"}
Sep 18 19:21:35.433240 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:35.432986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab","Type":"ContainerStarted","Data":"25e5345b4b69b996ecfe0f3bb5609af0f0f065e44094c443423b1ca4da385c93"}
Sep 18 19:21:36.440275 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:36.440241    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab","Type":"ContainerStarted","Data":"e82be43efde27ff71a201f1d5b7786e842156b3e82f23820caba6d5595e41fd6"}
Sep 18 19:21:36.440630 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:36.440302    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab","Type":"ContainerStarted","Data":"14dbc7c23143be6689671a0ccfc66ed0e5387e0dd6bd21c14183c2c854d84226"}
Sep 18 19:21:36.440630 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:36.440327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bbbeb7c-05f6-45f1-9a03-9e74ac903bab","Type":"ContainerStarted","Data":"42076952ce52bad8910a2a2c344b53993b91f1f15572dd012b39b14df7e1d3e4"}
Sep 18 19:21:36.489386 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:36.489330    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.668585274 podStartE2EDuration="6.489311233s" totalImagesPullingTime="2.797414814s" totalInitContainerRuntime="23.311145ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:21:36.487803707 +0000 UTC m=+418.941941883" watchObservedRunningTime="2026-09-18 19:21:36.489311233 +0000 UTC m=+418.943449410"
Sep 18 19:21:40.849391 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:40.849351    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:21:43.463903 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:21:43.463869    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78d5965bf4-9l5g9"]
Sep 18 19:22:08.484817 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.484732    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78d5965bf4-9l5g9" podUID="741aae46-ad5c-498b-a312-46a0e643968e" containerName="console" containerID="cri-o://66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0" gracePeriod=15
Sep 18 19:22:08.727667 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.727639    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:22:08.851328 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851298    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-console-config\") pod \"741aae46-ad5c-498b-a312-46a0e643968e\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") "
Sep 18 19:22:08.851328 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851331    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-service-ca\") pod \"741aae46-ad5c-498b-a312-46a0e643968e\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") "
Sep 18 19:22:08.851560 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851369    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-trusted-ca-bundle\") pod \"741aae46-ad5c-498b-a312-46a0e643968e\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") "
Sep 18 19:22:08.851560 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-oauth-serving-cert\") pod \"741aae46-ad5c-498b-a312-46a0e643968e\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") "
Sep 18 19:22:08.851560 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851431    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-serving-cert\") pod \"741aae46-ad5c-498b-a312-46a0e643968e\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") "
Sep 18 19:22:08.851560 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851479    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-oauth-config\") pod \"741aae46-ad5c-498b-a312-46a0e643968e\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") "
Sep 18 19:22:08.851560 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851524    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9wjq\" (UniqueName: \"kubernetes.io/projected/741aae46-ad5c-498b-a312-46a0e643968e-kube-api-access-t9wjq\") pod \"741aae46-ad5c-498b-a312-46a0e643968e\" (UID: \"741aae46-ad5c-498b-a312-46a0e643968e\") "
Sep 18 19:22:08.851813 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851715    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-console-config" (OuterVolumeSpecName: "console-config") pod "741aae46-ad5c-498b-a312-46a0e643968e" (UID: "741aae46-ad5c-498b-a312-46a0e643968e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:22:08.851813 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851807    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-console-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:22:08.851923 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851812    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-service-ca" (OuterVolumeSpecName: "service-ca") pod "741aae46-ad5c-498b-a312-46a0e643968e" (UID: "741aae46-ad5c-498b-a312-46a0e643968e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:22:08.851923 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851837    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "741aae46-ad5c-498b-a312-46a0e643968e" (UID: "741aae46-ad5c-498b-a312-46a0e643968e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:22:08.851923 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.851898    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "741aae46-ad5c-498b-a312-46a0e643968e" (UID: "741aae46-ad5c-498b-a312-46a0e643968e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:22:08.853851 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.853819    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "741aae46-ad5c-498b-a312-46a0e643968e" (UID: "741aae46-ad5c-498b-a312-46a0e643968e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:22:08.853851 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.853833    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741aae46-ad5c-498b-a312-46a0e643968e-kube-api-access-t9wjq" (OuterVolumeSpecName: "kube-api-access-t9wjq") pod "741aae46-ad5c-498b-a312-46a0e643968e" (UID: "741aae46-ad5c-498b-a312-46a0e643968e"). InnerVolumeSpecName "kube-api-access-t9wjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:22:08.854126 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.854105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "741aae46-ad5c-498b-a312-46a0e643968e" (UID: "741aae46-ad5c-498b-a312-46a0e643968e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:22:08.952613 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.952583    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-trusted-ca-bundle\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:22:08.952613 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.952606    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-oauth-serving-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:22:08.952762 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.952620    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-serving-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:22:08.952762 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.952646    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/741aae46-ad5c-498b-a312-46a0e643968e-console-oauth-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:22:08.952762 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.952654    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9wjq\" (UniqueName: \"kubernetes.io/projected/741aae46-ad5c-498b-a312-46a0e643968e-kube-api-access-t9wjq\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:22:08.952762 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:08.952662    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/741aae46-ad5c-498b-a312-46a0e643968e-service-ca\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:22:09.572564 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:09.572529    2559 generic.go:356] "Generic (PLEG): container finished" podID="741aae46-ad5c-498b-a312-46a0e643968e" containerID="66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0" exitCode=0
Sep 18 19:22:09.572564 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:09.572569    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d5965bf4-9l5g9"
Sep 18 19:22:09.573063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:09.572612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d5965bf4-9l5g9" event={"ID":"741aae46-ad5c-498b-a312-46a0e643968e","Type":"ContainerDied","Data":"66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0"}
Sep 18 19:22:09.573063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:09.572668    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d5965bf4-9l5g9" event={"ID":"741aae46-ad5c-498b-a312-46a0e643968e","Type":"ContainerDied","Data":"a5772925e402d1e010bb1ca459a929624a1f9b39ddde804d38b1d2f38fa82708"}
Sep 18 19:22:09.573063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:09.572700    2559 scope.go:117] "RemoveContainer" containerID="66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0"
Sep 18 19:22:09.581609 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:09.581593    2559 scope.go:117] "RemoveContainer" containerID="66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0"
Sep 18 19:22:09.581875 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:22:09.581854    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0\": container with ID starting with 66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0 not found: ID does not exist" containerID="66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0"
Sep 18 19:22:09.581953 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:09.581881    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0"} err="failed to get container status \"66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0\": rpc error: code = NotFound desc = could not find container \"66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0\": container with ID starting with 66ac2dbaa10cd16f93aa8b142457ef65cf0c607621549b8f80a7cc8171f899a0 not found: ID does not exist"
Sep 18 19:22:09.628476 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:09.628441    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78d5965bf4-9l5g9"]
Sep 18 19:22:09.637267 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:09.637242    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78d5965bf4-9l5g9"]
Sep 18 19:22:09.711096 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:09.711067    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="741aae46-ad5c-498b-a312-46a0e643968e" path="/var/lib/kubelet/pods/741aae46-ad5c-498b-a312-46a0e643968e/volumes"
Sep 18 19:22:30.850355 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:30.850318    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:22:30.866188 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:30.866161    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:22:31.685387 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:22:31.685362    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:24:39.625749 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:24:39.625724    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:24:39.628107 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:24:39.628086    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:28:50.763906 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.763874    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"]
Sep 18 19:28:50.764373 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.764234    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741aae46-ad5c-498b-a312-46a0e643968e" containerName="console"
Sep 18 19:28:50.764373 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.764247    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="741aae46-ad5c-498b-a312-46a0e643968e" containerName="console"
Sep 18 19:28:50.764373 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.764316    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="741aae46-ad5c-498b-a312-46a0e643968e" containerName="console"
Sep 18 19:28:50.767611 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.767595    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.772061 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.772035    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:28:50.772224 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.772036    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 19:28:50.772224 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.772039    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:28:50.772224 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.772150    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ct9wb\""
Sep 18 19:28:50.772405 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.772369    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 19:28:50.772634 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.772557    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:28:50.772761 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.772747    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:28:50.781258 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.781238    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"]
Sep 18 19:28:50.871392 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.871368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25227ebf-6b1f-4d73-bb9a-18e2de896172-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.871533 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.871409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.871533 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.871436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.871533 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.871493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25227ebf-6b1f-4d73-bb9a-18e2de896172-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.871533 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.871525    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxq72\" (UniqueName: \"kubernetes.io/projected/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-api-access-qxq72\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.972088 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.972056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.972257 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.972103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25227ebf-6b1f-4d73-bb9a-18e2de896172-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.972257 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.972137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxq72\" (UniqueName: \"kubernetes.io/projected/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-api-access-qxq72\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.972257 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.972188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25227ebf-6b1f-4d73-bb9a-18e2de896172-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.972257 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.972240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.972658 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.972622    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25227ebf-6b1f-4d73-bb9a-18e2de896172-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.972834 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.972811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.972894 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.972831    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.974523 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.974498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25227ebf-6b1f-4d73-bb9a-18e2de896172-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:50.980345 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:50.980323    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxq72\" (UniqueName: \"kubernetes.io/projected/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-api-access-qxq72\") pod \"isvc-canary-rollback-predictor-794fd55d6c-b8p7p\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:51.078550 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:51.078521    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:28:51.355159 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:51.355129    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"]
Sep 18 19:28:51.356420 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:28:51.356397    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25227ebf_6b1f_4d73_bb9a_18e2de896172.slice/crio-e0af50386beeb5eba89efe5c13383c64c2d6078c9b0c973b56a9f94defa2dbc2 WatchSource:0}: Error finding container e0af50386beeb5eba89efe5c13383c64c2d6078c9b0c973b56a9f94defa2dbc2: Status 404 returned error can't find the container with id e0af50386beeb5eba89efe5c13383c64c2d6078c9b0c973b56a9f94defa2dbc2
Sep 18 19:28:51.358575 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:51.358559    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:28:52.187965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:52.187926    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" event={"ID":"25227ebf-6b1f-4d73-bb9a-18e2de896172","Type":"ContainerStarted","Data":"e0af50386beeb5eba89efe5c13383c64c2d6078c9b0c973b56a9f94defa2dbc2"}
Sep 18 19:28:55.201963 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:55.201879    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" event={"ID":"25227ebf-6b1f-4d73-bb9a-18e2de896172","Type":"ContainerStarted","Data":"e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255"}
Sep 18 19:28:59.221021 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:59.220940    2559 generic.go:356] "Generic (PLEG): container finished" podID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerID="e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255" exitCode=0
Sep 18 19:28:59.221432 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:28:59.221013    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" event={"ID":"25227ebf-6b1f-4d73-bb9a-18e2de896172","Type":"ContainerDied","Data":"e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255"}
Sep 18 19:29:13.298722 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:13.298686    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" event={"ID":"25227ebf-6b1f-4d73-bb9a-18e2de896172","Type":"ContainerStarted","Data":"81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9"}
Sep 18 19:29:15.308710 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:15.308683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" event={"ID":"25227ebf-6b1f-4d73-bb9a-18e2de896172","Type":"ContainerStarted","Data":"2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7"}
Sep 18 19:29:15.308993 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:15.308757    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:29:15.336329 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:15.336289    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podStartSLOduration=2.049786648 podStartE2EDuration="25.336274401s" totalImagesPullingTime="19.466524004s" totalInitContainerRuntime="3.819963749s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:28:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:29:15.334858529 +0000 UTC m=+877.788996706" watchObservedRunningTime="2026-09-18 19:29:15.336274401 +0000 UTC m=+877.790412578"
Sep 18 19:29:16.312582 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:16.312546    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:29:16.313802 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:16.313766    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:29:17.316066 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:17.316033    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:29:22.320852 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:22.320775    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:29:22.321330 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:22.321303    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:29:32.321253 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:32.321193    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:29:39.657550 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:39.657519    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:29:39.658859 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:39.658837    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:29:42.321432 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:42.321395    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:29:52.321539 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:29:52.321496    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:30:02.321735 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:02.321699    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:30:12.321993 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:12.321953    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:30:22.321842 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:22.321804    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:30:32.322369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:32.322340    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:30:46.235184 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.235152    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"]
Sep 18 19:30:46.237817 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.237796    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.243050 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.242989    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-4a391-kube-rbac-proxy-sar-config\""
Sep 18 19:30:46.243725 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.243705    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-4a391-predictor-serving-cert\""
Sep 18 19:30:46.259819 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.259797    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"]
Sep 18 19:30:46.351905 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.351873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb55976-52e2-417e-be5e-eaec9bb51fc3-proxy-tls\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.352063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.351912    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.352063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.351972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.352063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.352032    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwqct\" (UniqueName: \"kubernetes.io/projected/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-api-access-kwqct\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.352063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.352059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.453120 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.453072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb55976-52e2-417e-be5e-eaec9bb51fc3-proxy-tls\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.453120 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.453125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.453385 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.453157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.453385 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.453222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwqct\" (UniqueName: \"kubernetes.io/projected/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-api-access-kwqct\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.453385 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.453248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.453711 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.453679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.453909 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.453889    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.453975 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.453934    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.455612 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.455593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb55976-52e2-417e-be5e-eaec9bb51fc3-proxy-tls\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.462840 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.462817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwqct\" (UniqueName: \"kubernetes.io/projected/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-api-access-kwqct\") pod \"isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.547666 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.547629    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:30:46.827797 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:46.827772    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"]
Sep 18 19:30:46.830576 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:30:46.830547    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb55976_52e2_417e_be5e_eaec9bb51fc3.slice/crio-02e6d53aa61c0d95c47a02eb4d61b35a8ef52520d86aa3bca97ded9ff1bd380c WatchSource:0}: Error finding container 02e6d53aa61c0d95c47a02eb4d61b35a8ef52520d86aa3bca97ded9ff1bd380c: Status 404 returned error can't find the container with id 02e6d53aa61c0d95c47a02eb4d61b35a8ef52520d86aa3bca97ded9ff1bd380c
Sep 18 19:30:47.683475 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:47.683434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" event={"ID":"cdb55976-52e2-417e-be5e-eaec9bb51fc3","Type":"ContainerStarted","Data":"5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c"}
Sep 18 19:30:47.683936 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:47.683516    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" event={"ID":"cdb55976-52e2-417e-be5e-eaec9bb51fc3","Type":"ContainerStarted","Data":"02e6d53aa61c0d95c47a02eb4d61b35a8ef52520d86aa3bca97ded9ff1bd380c"}
Sep 18 19:30:50.698122 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:50.698038    2559 generic.go:356] "Generic (PLEG): container finished" podID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerID="5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c" exitCode=0
Sep 18 19:30:50.698122 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:30:50.698097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" event={"ID":"cdb55976-52e2-417e-be5e-eaec9bb51fc3","Type":"ContainerDied","Data":"5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c"}
Sep 18 19:31:10.798054 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:10.798017    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" event={"ID":"cdb55976-52e2-417e-be5e-eaec9bb51fc3","Type":"ContainerStarted","Data":"6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284"}
Sep 18 19:31:10.798541 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:10.798091    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" event={"ID":"cdb55976-52e2-417e-be5e-eaec9bb51fc3","Type":"ContainerStarted","Data":"2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89"}
Sep 18 19:31:10.798541 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:10.798346    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:31:10.824585 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:10.824542    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podStartSLOduration=1.408865795 podStartE2EDuration="24.824527522s" totalImagesPullingTime="19.90784808s" totalInitContainerRuntime="3.507813647s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:31:10.822659363 +0000 UTC m=+993.276797543" watchObservedRunningTime="2026-09-18 19:31:10.824527522 +0000 UTC m=+993.278665698"
Sep 18 19:31:11.802448 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:11.802409    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:31:11.803872 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:11.803839    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:31:12.806170 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:12.806132    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:31:17.810115 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:17.810089    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:31:17.810606 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:17.810547    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:31:27.810997 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:27.810947    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:31:37.810736 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:37.810701    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:31:47.810765 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:47.810725    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:31:57.810541 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:31:57.810501    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:32:07.811508 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:07.811466    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:32:10.480185 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:10.480149    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"]
Sep 18 19:32:10.480717 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:10.480418    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" containerID="cri-o://81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9" gracePeriod=30
Sep 18 19:32:10.480717 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:10.480479    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kube-rbac-proxy" containerID="cri-o://2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7" gracePeriod=30
Sep 18 19:32:11.043803 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:11.043770    2559 generic.go:356] "Generic (PLEG): container finished" podID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerID="2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7" exitCode=2
Sep 18 19:32:11.043980 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:11.043841    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" event={"ID":"25227ebf-6b1f-4d73-bb9a-18e2de896172","Type":"ContainerDied","Data":"2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7"}
Sep 18 19:32:12.316889 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:12.316848    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 18 19:32:12.321212 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:12.321178    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:32:15.423123 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.423101    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:32:15.537599 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.537570    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-cabundle-cert\") pod \"25227ebf-6b1f-4d73-bb9a-18e2de896172\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") "
Sep 18 19:32:15.537749 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.537629    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25227ebf-6b1f-4d73-bb9a-18e2de896172-proxy-tls\") pod \"25227ebf-6b1f-4d73-bb9a-18e2de896172\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") "
Sep 18 19:32:15.537749 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.537691    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-rbac-proxy-sar-config\") pod \"25227ebf-6b1f-4d73-bb9a-18e2de896172\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") "
Sep 18 19:32:15.537869 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.537746    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxq72\" (UniqueName: \"kubernetes.io/projected/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-api-access-qxq72\") pod \"25227ebf-6b1f-4d73-bb9a-18e2de896172\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") "
Sep 18 19:32:15.537869 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.537778    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25227ebf-6b1f-4d73-bb9a-18e2de896172-kserve-provision-location\") pod \"25227ebf-6b1f-4d73-bb9a-18e2de896172\" (UID: \"25227ebf-6b1f-4d73-bb9a-18e2de896172\") "
Sep 18 19:32:15.538071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.538044    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "25227ebf-6b1f-4d73-bb9a-18e2de896172" (UID: "25227ebf-6b1f-4d73-bb9a-18e2de896172"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:32:15.538141 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.538076    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "25227ebf-6b1f-4d73-bb9a-18e2de896172" (UID: "25227ebf-6b1f-4d73-bb9a-18e2de896172"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:32:15.538141 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.538103    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25227ebf-6b1f-4d73-bb9a-18e2de896172-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25227ebf-6b1f-4d73-bb9a-18e2de896172" (UID: "25227ebf-6b1f-4d73-bb9a-18e2de896172"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:32:15.539842 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.539821    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25227ebf-6b1f-4d73-bb9a-18e2de896172-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25227ebf-6b1f-4d73-bb9a-18e2de896172" (UID: "25227ebf-6b1f-4d73-bb9a-18e2de896172"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:32:15.539947 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.539930    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-api-access-qxq72" (OuterVolumeSpecName: "kube-api-access-qxq72") pod "25227ebf-6b1f-4d73-bb9a-18e2de896172" (UID: "25227ebf-6b1f-4d73-bb9a-18e2de896172"). InnerVolumeSpecName "kube-api-access-qxq72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:32:15.639109 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.639078    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxq72\" (UniqueName: \"kubernetes.io/projected/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-api-access-qxq72\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:32:15.639109 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.639108    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25227ebf-6b1f-4d73-bb9a-18e2de896172-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:32:15.639325 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.639121    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:32:15.639325 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.639134    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25227ebf-6b1f-4d73-bb9a-18e2de896172-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:32:15.639325 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.639146    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25227ebf-6b1f-4d73-bb9a-18e2de896172-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:32:15.851638 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.851545    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"]
Sep 18 19:32:15.852097 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.852073    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container"
Sep 18 19:32:15.852097 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.852097    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container"
Sep 18 19:32:15.852288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.852111    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kube-rbac-proxy"
Sep 18 19:32:15.852288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.852120    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kube-rbac-proxy"
Sep 18 19:32:15.852288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.852133    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="storage-initializer"
Sep 18 19:32:15.852288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.852141    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="storage-initializer"
Sep 18 19:32:15.852288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.852278    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kube-rbac-proxy"
Sep 18 19:32:15.852527 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.852300    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerName="kserve-container"
Sep 18 19:32:15.855323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.855299    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:15.858876 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.858848    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 19:32:15.859022 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.858992    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 19:32:15.871942 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.871915    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"]
Sep 18 19:32:15.941930 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.941898    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:15.942060 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.941941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b93c78-9094-4b1c-8e40-0dfb10580d71-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:15.942114 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.942086    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b93c78-9094-4b1c-8e40-0dfb10580d71-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:15.942156 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.942113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnsc2\" (UniqueName: \"kubernetes.io/projected/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-api-access-nnsc2\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:15.942156 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:15.942131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.042857 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.042829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b93c78-9094-4b1c-8e40-0dfb10580d71-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.042857 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.042860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsc2\" (UniqueName: \"kubernetes.io/projected/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-api-access-nnsc2\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.043082 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.042876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.043082 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.042895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.043082 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.042922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b93c78-9094-4b1c-8e40-0dfb10580d71-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.043421 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.043401    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b93c78-9094-4b1c-8e40-0dfb10580d71-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.043606 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.043588    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.043788 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.043764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.045488 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.045472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b93c78-9094-4b1c-8e40-0dfb10580d71-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.050894 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.050877    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsc2\" (UniqueName: \"kubernetes.io/projected/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-api-access-nnsc2\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.067951 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.067924    2559 generic.go:356] "Generic (PLEG): container finished" podID="25227ebf-6b1f-4d73-bb9a-18e2de896172" containerID="81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9" exitCode=0
Sep 18 19:32:16.068038 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.067968    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" event={"ID":"25227ebf-6b1f-4d73-bb9a-18e2de896172","Type":"ContainerDied","Data":"81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9"}
Sep 18 19:32:16.068038 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.067994    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"
Sep 18 19:32:16.068038 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.068020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p" event={"ID":"25227ebf-6b1f-4d73-bb9a-18e2de896172","Type":"ContainerDied","Data":"e0af50386beeb5eba89efe5c13383c64c2d6078c9b0c973b56a9f94defa2dbc2"}
Sep 18 19:32:16.068217 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.068047    2559 scope.go:117] "RemoveContainer" containerID="2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7"
Sep 18 19:32:16.076358 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.076340    2559 scope.go:117] "RemoveContainer" containerID="81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9"
Sep 18 19:32:16.083700 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.083678    2559 scope.go:117] "RemoveContainer" containerID="e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255"
Sep 18 19:32:16.085490 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.085472    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"]
Sep 18 19:32:16.093223 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.091349    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-b8p7p"]
Sep 18 19:32:16.093223 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.091662    2559 scope.go:117] "RemoveContainer" containerID="2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7"
Sep 18 19:32:16.096399 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:32:16.096370    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7\": container with ID starting with 2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7 not found: ID does not exist" containerID="2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7"
Sep 18 19:32:16.096505 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.096402    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7"} err="failed to get container status \"2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7\": rpc error: code = NotFound desc = could not find container \"2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7\": container with ID starting with 2e0bb864377e55b3d74c75202ca41a131f7c2690599d0ed6393811b6425e24a7 not found: ID does not exist"
Sep 18 19:32:16.096505 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.096419    2559 scope.go:117] "RemoveContainer" containerID="81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9"
Sep 18 19:32:16.096702 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:32:16.096685    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9\": container with ID starting with 81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9 not found: ID does not exist" containerID="81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9"
Sep 18 19:32:16.096760 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.096706    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9"} err="failed to get container status \"81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9\": rpc error: code = NotFound desc = could not find container \"81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9\": container with ID starting with 81673e3fe05ba3a613f2df547969584273201b1554b23f2bd1721175d3b181c9 not found: ID does not exist"
Sep 18 19:32:16.096760 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.096719    2559 scope.go:117] "RemoveContainer" containerID="e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255"
Sep 18 19:32:16.096989 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:32:16.096969    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255\": container with ID starting with e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255 not found: ID does not exist" containerID="e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255"
Sep 18 19:32:16.097030 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.096994    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255"} err="failed to get container status \"e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255\": rpc error: code = NotFound desc = could not find container \"e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255\": container with ID starting with e016598f3ed483404a7042c5a8665b9b139ee198575ac25016d8cbc9bfab1255 not found: ID does not exist"
Sep 18 19:32:16.166776 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.166699    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:16.445131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:16.445102    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"]
Sep 18 19:32:16.446819 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:32:16.446784    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b93c78_9094_4b1c_8e40_0dfb10580d71.slice/crio-ac00110533edb49f231f0bde7cde584ee30d7fcf4b0523f50fa527ac8f82e54f WatchSource:0}: Error finding container ac00110533edb49f231f0bde7cde584ee30d7fcf4b0523f50fa527ac8f82e54f: Status 404 returned error can't find the container with id ac00110533edb49f231f0bde7cde584ee30d7fcf4b0523f50fa527ac8f82e54f
Sep 18 19:32:17.075366 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:17.075330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" event={"ID":"35b93c78-9094-4b1c-8e40-0dfb10580d71","Type":"ContainerStarted","Data":"6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9"}
Sep 18 19:32:17.075536 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:17.075400    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" event={"ID":"35b93c78-9094-4b1c-8e40-0dfb10580d71","Type":"ContainerStarted","Data":"ac00110533edb49f231f0bde7cde584ee30d7fcf4b0523f50fa527ac8f82e54f"}
Sep 18 19:32:17.710770 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:17.710742    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25227ebf-6b1f-4d73-bb9a-18e2de896172" path="/var/lib/kubelet/pods/25227ebf-6b1f-4d73-bb9a-18e2de896172/volumes"
Sep 18 19:32:17.811113 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:17.811083    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:32:21.093818 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:21.093788    2559 generic.go:356] "Generic (PLEG): container finished" podID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerID="6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9" exitCode=0
Sep 18 19:32:21.094229 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:21.093861    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" event={"ID":"35b93c78-9094-4b1c-8e40-0dfb10580d71","Type":"ContainerDied","Data":"6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9"}
Sep 18 19:32:22.100105 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:22.100071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" event={"ID":"35b93c78-9094-4b1c-8e40-0dfb10580d71","Type":"ContainerStarted","Data":"6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f"}
Sep 18 19:32:22.100480 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:22.100132    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" event={"ID":"35b93c78-9094-4b1c-8e40-0dfb10580d71","Type":"ContainerStarted","Data":"e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7"}
Sep 18 19:32:22.100480 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:22.100333    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:22.128260 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:22.128180    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podStartSLOduration=3.491424866 podStartE2EDuration="7.128167001s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.636742135s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:32:22.127590228 +0000 UTC m=+1064.581728406" watchObservedRunningTime="2026-09-18 19:32:22.128167001 +0000 UTC m=+1064.582305177"
Sep 18 19:32:23.103965 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:23.103931    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:23.105276 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:23.105249    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:32:24.107958 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:24.107921    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:32:29.113993 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:29.113966    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:32:29.114407 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:29.114377    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:32:36.289166 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.289134    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"]
Sep 18 19:32:36.289598 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.289400    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container" containerID="cri-o://2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89" gracePeriod=30
Sep 18 19:32:36.289598 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.289423    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kube-rbac-proxy" containerID="cri-o://6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284" gracePeriod=30
Sep 18 19:32:36.484093 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.484060    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"]
Sep 18 19:32:36.486674 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.486659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.492344 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.492322    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-db55d-kube-rbac-proxy-sar-config\""
Sep 18 19:32:36.492477 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.492340    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-db55d-predictor-serving-cert\""
Sep 18 19:32:36.507815 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.507789    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"]
Sep 18 19:32:36.603373 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.603299    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"]
Sep 18 19:32:36.605923 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.605903    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.610823 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.610804    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-db55d-predictor-serving-cert\""
Sep 18 19:32:36.610949 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.610934    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-db55d-kube-rbac-proxy-sar-config\""
Sep 18 19:32:36.615494 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.615474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.615588 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.615513    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.615588 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.615546    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq96t\" (UniqueName: \"kubernetes.io/projected/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-api-access-sq96t\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.615588 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.615585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.615702 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.615637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.624479 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.624458    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"]
Sep 18 19:32:36.716230 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716190    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.716361 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.716361 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716321    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.716361 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.716576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716385    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707c46b7-8078-4090-9148-53056998e846-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.716576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.716576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchq2\" (UniqueName: \"kubernetes.io/projected/707c46b7-8078-4090-9148-53056998e846-kube-api-access-tchq2\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.716576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707c46b7-8078-4090-9148-53056998e846-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.716576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.716828 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq96t\" (UniqueName: \"kubernetes.io/projected/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-api-access-sq96t\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.716885 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.716855    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.717063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.717042    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.717124 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.717066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.718553 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.718532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.725353 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.725332    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq96t\" (UniqueName: \"kubernetes.io/projected/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-api-access-sq96t\") pod \"isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.798022 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.797993    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:36.817299 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.817263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchq2\" (UniqueName: \"kubernetes.io/projected/707c46b7-8078-4090-9148-53056998e846-kube-api-access-tchq2\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.817392 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.817319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707c46b7-8078-4090-9148-53056998e846-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.817392 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.817368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.817527 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.817462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.817527 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.817504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707c46b7-8078-4090-9148-53056998e846-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.817880 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:32:36.817793    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-db55d-predictor-serving-cert" not found
Sep 18 19:32:36.817880 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:32:36.817877    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/707c46b7-8078-4090-9148-53056998e846-proxy-tls podName:707c46b7-8078-4090-9148-53056998e846 nodeName:}" failed. No retries permitted until 2026-09-18 19:32:37.317855584 +0000 UTC m=+1079.771993748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/707c46b7-8078-4090-9148-53056998e846-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" (UID: "707c46b7-8078-4090-9148-53056998e846") : secret "isvc-xgboost-graph-raw-hpa-db55d-predictor-serving-cert" not found
Sep 18 19:32:36.818084 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.817898    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707c46b7-8078-4090-9148-53056998e846-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.818334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.818312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.818406 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.818338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:36.827735 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:36.827704    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchq2\" (UniqueName: \"kubernetes.io/projected/707c46b7-8078-4090-9148-53056998e846-kube-api-access-tchq2\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:37.081037 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.081014    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"]
Sep 18 19:32:37.083521 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:32:37.083491    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be91c6e_819c_4c2a_a7c1_9eb7085da62c.slice/crio-521e2e22c804de6f1b4483420fe33cd857859605c965446d9e8ffc3abf16097e WatchSource:0}: Error finding container 521e2e22c804de6f1b4483420fe33cd857859605c965446d9e8ffc3abf16097e: Status 404 returned error can't find the container with id 521e2e22c804de6f1b4483420fe33cd857859605c965446d9e8ffc3abf16097e
Sep 18 19:32:37.167946 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.167914    2559 generic.go:356] "Generic (PLEG): container finished" podID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerID="6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284" exitCode=2
Sep 18 19:32:37.168085 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.167990    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" event={"ID":"cdb55976-52e2-417e-be5e-eaec9bb51fc3","Type":"ContainerDied","Data":"6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284"}
Sep 18 19:32:37.169725 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.169701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" event={"ID":"0be91c6e-819c-4c2a-a7c1-9eb7085da62c","Type":"ContainerStarted","Data":"a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e"}
Sep 18 19:32:37.169822 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.169755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" event={"ID":"0be91c6e-819c-4c2a-a7c1-9eb7085da62c","Type":"ContainerStarted","Data":"521e2e22c804de6f1b4483420fe33cd857859605c965446d9e8ffc3abf16097e"}
Sep 18 19:32:37.323323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.323288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707c46b7-8078-4090-9148-53056998e846-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:37.325625 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.325601    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707c46b7-8078-4090-9148-53056998e846-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:37.515416 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.515336    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:37.806743 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.806693    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 19:32:37.809269 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.809249    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"]
Sep 18 19:32:37.810543 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:37.810513    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:32:37.810711 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:32:37.810690    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707c46b7_8078_4090_9148_53056998e846.slice/crio-c9abca681ce7df0770eb3a003793c391dbd8efaeebf9d3904e3ba3034d71317b WatchSource:0}: Error finding container c9abca681ce7df0770eb3a003793c391dbd8efaeebf9d3904e3ba3034d71317b: Status 404 returned error can't find the container with id c9abca681ce7df0770eb3a003793c391dbd8efaeebf9d3904e3ba3034d71317b
Sep 18 19:32:38.176268 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:38.176173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" event={"ID":"707c46b7-8078-4090-9148-53056998e846","Type":"ContainerStarted","Data":"ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1"}
Sep 18 19:32:38.176268 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:38.176242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" event={"ID":"707c46b7-8078-4090-9148-53056998e846","Type":"ContainerStarted","Data":"c9abca681ce7df0770eb3a003793c391dbd8efaeebf9d3904e3ba3034d71317b"}
Sep 18 19:32:39.114859 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:39.114819    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:32:40.731569 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.731527    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:32:40.854937 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.854846    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwqct\" (UniqueName: \"kubernetes.io/projected/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-api-access-kwqct\") pod \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") "
Sep 18 19:32:40.854937 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.854886    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-rbac-proxy-sar-config\") pod \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") "
Sep 18 19:32:40.854937 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.854926    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb55976-52e2-417e-be5e-eaec9bb51fc3-proxy-tls\") pod \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") "
Sep 18 19:32:40.855229 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.855044    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kserve-provision-location\") pod \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") "
Sep 18 19:32:40.855229 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.855108    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-cabundle-cert\") pod \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\" (UID: \"cdb55976-52e2-417e-be5e-eaec9bb51fc3\") "
Sep 18 19:32:40.855515 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.855429    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cdb55976-52e2-417e-be5e-eaec9bb51fc3" (UID: "cdb55976-52e2-417e-be5e-eaec9bb51fc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:32:40.855641 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.855613    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cdb55976-52e2-417e-be5e-eaec9bb51fc3" (UID: "cdb55976-52e2-417e-be5e-eaec9bb51fc3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:32:40.855720 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.855703    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cdb55976-52e2-417e-be5e-eaec9bb51fc3" (UID: "cdb55976-52e2-417e-be5e-eaec9bb51fc3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:32:40.857082 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.857049    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb55976-52e2-417e-be5e-eaec9bb51fc3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cdb55976-52e2-417e-be5e-eaec9bb51fc3" (UID: "cdb55976-52e2-417e-be5e-eaec9bb51fc3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:32:40.857246 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.857139    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-api-access-kwqct" (OuterVolumeSpecName: "kube-api-access-kwqct") pod "cdb55976-52e2-417e-be5e-eaec9bb51fc3" (UID: "cdb55976-52e2-417e-be5e-eaec9bb51fc3"). InnerVolumeSpecName "kube-api-access-kwqct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:32:40.956349 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.956317    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwqct\" (UniqueName: \"kubernetes.io/projected/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-api-access-kwqct\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:32:40.956349 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.956342    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:32:40.956349 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.956354    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb55976-52e2-417e-be5e-eaec9bb51fc3-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:32:40.956562 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.956364    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdb55976-52e2-417e-be5e-eaec9bb51fc3-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:32:40.956562 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:40.956372    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdb55976-52e2-417e-be5e-eaec9bb51fc3-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:32:41.191436 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.191358    2559 generic.go:356] "Generic (PLEG): container finished" podID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerID="2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89" exitCode=0
Sep 18 19:32:41.191436 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.191418    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"
Sep 18 19:32:41.191620 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.191433    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" event={"ID":"cdb55976-52e2-417e-be5e-eaec9bb51fc3","Type":"ContainerDied","Data":"2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89"}
Sep 18 19:32:41.191620 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.191544    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2" event={"ID":"cdb55976-52e2-417e-be5e-eaec9bb51fc3","Type":"ContainerDied","Data":"02e6d53aa61c0d95c47a02eb4d61b35a8ef52520d86aa3bca97ded9ff1bd380c"}
Sep 18 19:32:41.191620 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.191589    2559 scope.go:117] "RemoveContainer" containerID="6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284"
Sep 18 19:32:41.193102 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.193078    2559 generic.go:356] "Generic (PLEG): container finished" podID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerID="a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e" exitCode=0
Sep 18 19:32:41.193266 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.193131    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" event={"ID":"0be91c6e-819c-4c2a-a7c1-9eb7085da62c","Type":"ContainerDied","Data":"a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e"}
Sep 18 19:32:41.200300 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.200282    2559 scope.go:117] "RemoveContainer" containerID="2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89"
Sep 18 19:32:41.207716 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.207700    2559 scope.go:117] "RemoveContainer" containerID="5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c"
Sep 18 19:32:41.216030 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.216013    2559 scope.go:117] "RemoveContainer" containerID="6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284"
Sep 18 19:32:41.216293 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:32:41.216272    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284\": container with ID starting with 6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284 not found: ID does not exist" containerID="6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284"
Sep 18 19:32:41.216371 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.216305    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284"} err="failed to get container status \"6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284\": rpc error: code = NotFound desc = could not find container \"6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284\": container with ID starting with 6457fec61c92631235dc659af61b597169c060c8a370917fb31239bfbbcfb284 not found: ID does not exist"
Sep 18 19:32:41.216371 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.216327    2559 scope.go:117] "RemoveContainer" containerID="2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89"
Sep 18 19:32:41.219027 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:32:41.218997    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89\": container with ID starting with 2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89 not found: ID does not exist" containerID="2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89"
Sep 18 19:32:41.219136 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.219031    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89"} err="failed to get container status \"2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89\": rpc error: code = NotFound desc = could not find container \"2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89\": container with ID starting with 2913e257e3ba6b4abc4b8b5f8673becbeac1132600d984399f00f24b21907f89 not found: ID does not exist"
Sep 18 19:32:41.219136 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.219047    2559 scope.go:117] "RemoveContainer" containerID="5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c"
Sep 18 19:32:41.219777 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:32:41.219754    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c\": container with ID starting with 5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c not found: ID does not exist" containerID="5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c"
Sep 18 19:32:41.219872 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.219783    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c"} err="failed to get container status \"5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c\": rpc error: code = NotFound desc = could not find container \"5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c\": container with ID starting with 5a46a9b242d0034c0cff9ec522b8dc640a4accacf195b5fb6732a5b7086d585c not found: ID does not exist"
Sep 18 19:32:41.234932 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.234911    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"]
Sep 18 19:32:41.237804 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.237783    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4a391-predictor-577f6578d6-rjpj2"]
Sep 18 19:32:41.710865 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:41.710836    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" path="/var/lib/kubelet/pods/cdb55976-52e2-417e-be5e-eaec9bb51fc3/volumes"
Sep 18 19:32:42.200307 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:42.200218    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" event={"ID":"0be91c6e-819c-4c2a-a7c1-9eb7085da62c","Type":"ContainerStarted","Data":"a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c"}
Sep 18 19:32:42.200307 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:42.200285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" event={"ID":"0be91c6e-819c-4c2a-a7c1-9eb7085da62c","Type":"ContainerStarted","Data":"10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac"}
Sep 18 19:32:42.200747 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:42.200500    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:42.201730 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:42.201710    2559 generic.go:356] "Generic (PLEG): container finished" podID="707c46b7-8078-4090-9148-53056998e846" containerID="ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1" exitCode=0
Sep 18 19:32:42.201830 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:42.201765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" event={"ID":"707c46b7-8078-4090-9148-53056998e846","Type":"ContainerDied","Data":"ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1"}
Sep 18 19:32:42.225257 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:42.225189    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podStartSLOduration=2.359474725 podStartE2EDuration="6.225170352s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.865695627s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:32:42.223732561 +0000 UTC m=+1084.677870740" watchObservedRunningTime="2026-09-18 19:32:42.225170352 +0000 UTC m=+1084.679308529"
Sep 18 19:32:43.209083 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:43.209050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" event={"ID":"707c46b7-8078-4090-9148-53056998e846","Type":"ContainerStarted","Data":"74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742"}
Sep 18 19:32:43.209481 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:43.209314    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:43.209481 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:43.209346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" event={"ID":"707c46b7-8078-4090-9148-53056998e846","Type":"ContainerStarted","Data":"b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a"}
Sep 18 19:32:43.209671 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:43.209651    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:43.209817 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:43.209786    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:43.210708 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:43.210684    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:32:43.210800 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:43.210710    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:32:43.234820 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:43.234779    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podStartSLOduration=3.253102476 podStartE2EDuration="7.234759073s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.981656597s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:32:43.232897307 +0000 UTC m=+1085.687035506" watchObservedRunningTime="2026-09-18 19:32:43.234759073 +0000 UTC m=+1085.688897249"
Sep 18 19:32:44.214335 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:44.214297    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:32:44.214731 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:44.214441    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:32:45.218660 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:45.218623    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:32:49.115176 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:49.115140    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:32:49.219106 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:49.219073    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:32:49.219644 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:49.219614    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:32:50.224518 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:50.224490    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:32:50.224972 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:50.224950    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:32:59.115142 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:59.115101    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:32:59.219967 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:32:59.219929    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:33:00.225192 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:00.225155    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:33:09.114423 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:09.114387    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:33:09.220505 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:09.220465    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:33:10.225196 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:10.225159    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:33:19.114401 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:19.114364    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:33:19.220378 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:19.220337    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:33:20.225067 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:20.225030    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:33:29.115393 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:29.115349    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:33:29.220055 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:29.220012    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:33:30.225065 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:30.225025    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:33:38.858660 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:38.858627    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"]
Sep 18 19:33:38.859035 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:38.858925    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container" containerID="cri-o://e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7" gracePeriod=30
Sep 18 19:33:38.859541 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:38.859378    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kube-rbac-proxy" containerID="cri-o://6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f" gracePeriod=30
Sep 18 19:33:38.865845 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:38.865822    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:33:39.108388 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:39.108351    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 19:33:39.220562 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:39.220480    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:33:39.468032 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:39.468005    2559 generic.go:356] "Generic (PLEG): container finished" podID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerID="6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f" exitCode=2
Sep 18 19:33:39.468197 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:39.468083    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" event={"ID":"35b93c78-9094-4b1c-8e40-0dfb10580d71","Type":"ContainerDied","Data":"6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f"}
Sep 18 19:33:40.225460 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:40.225426    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:33:43.809990 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.809966    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:33:43.910050 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.909988    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnsc2\" (UniqueName: \"kubernetes.io/projected/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-api-access-nnsc2\") pod \"35b93c78-9094-4b1c-8e40-0dfb10580d71\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") "
Sep 18 19:33:43.910050 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.910032    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-rbac-proxy-sar-config\") pod \"35b93c78-9094-4b1c-8e40-0dfb10580d71\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") "
Sep 18 19:33:43.910255 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.910115    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b93c78-9094-4b1c-8e40-0dfb10580d71-kserve-provision-location\") pod \"35b93c78-9094-4b1c-8e40-0dfb10580d71\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") "
Sep 18 19:33:43.910255 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.910146    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-cabundle-cert\") pod \"35b93c78-9094-4b1c-8e40-0dfb10580d71\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") "
Sep 18 19:33:43.910255 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.910176    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b93c78-9094-4b1c-8e40-0dfb10580d71-proxy-tls\") pod \"35b93c78-9094-4b1c-8e40-0dfb10580d71\" (UID: \"35b93c78-9094-4b1c-8e40-0dfb10580d71\") "
Sep 18 19:33:43.910525 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.910501    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b93c78-9094-4b1c-8e40-0dfb10580d71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35b93c78-9094-4b1c-8e40-0dfb10580d71" (UID: "35b93c78-9094-4b1c-8e40-0dfb10580d71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:33:43.910587 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.910526    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35b93c78-9094-4b1c-8e40-0dfb10580d71" (UID: "35b93c78-9094-4b1c-8e40-0dfb10580d71"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:33:43.910627 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.910592    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35b93c78-9094-4b1c-8e40-0dfb10580d71" (UID: "35b93c78-9094-4b1c-8e40-0dfb10580d71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:33:43.912128 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.912105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-api-access-nnsc2" (OuterVolumeSpecName: "kube-api-access-nnsc2") pod "35b93c78-9094-4b1c-8e40-0dfb10580d71" (UID: "35b93c78-9094-4b1c-8e40-0dfb10580d71"). InnerVolumeSpecName "kube-api-access-nnsc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:33:43.912356 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:43.912331    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b93c78-9094-4b1c-8e40-0dfb10580d71-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35b93c78-9094-4b1c-8e40-0dfb10580d71" (UID: "35b93c78-9094-4b1c-8e40-0dfb10580d71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:33:44.010958 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.010919    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnsc2\" (UniqueName: \"kubernetes.io/projected/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-api-access-nnsc2\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:33:44.010958 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.010947    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:33:44.010958 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.010956    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b93c78-9094-4b1c-8e40-0dfb10580d71-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:33:44.010958 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.010965    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b93c78-9094-4b1c-8e40-0dfb10580d71-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:33:44.011235 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.010974    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b93c78-9094-4b1c-8e40-0dfb10580d71-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:33:44.494315 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.494284    2559 generic.go:356] "Generic (PLEG): container finished" podID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerID="e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7" exitCode=0
Sep 18 19:33:44.494315 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.494321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" event={"ID":"35b93c78-9094-4b1c-8e40-0dfb10580d71","Type":"ContainerDied","Data":"e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7"}
Sep 18 19:33:44.494315 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.494337    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"
Sep 18 19:33:44.494607 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.494363    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw" event={"ID":"35b93c78-9094-4b1c-8e40-0dfb10580d71","Type":"ContainerDied","Data":"ac00110533edb49f231f0bde7cde584ee30d7fcf4b0523f50fa527ac8f82e54f"}
Sep 18 19:33:44.494607 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.494390    2559 scope.go:117] "RemoveContainer" containerID="6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f"
Sep 18 19:33:44.503527 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.503510    2559 scope.go:117] "RemoveContainer" containerID="e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7"
Sep 18 19:33:44.511063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.511048    2559 scope.go:117] "RemoveContainer" containerID="6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9"
Sep 18 19:33:44.517682 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.517662    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"]
Sep 18 19:33:44.518384 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.518368    2559 scope.go:117] "RemoveContainer" containerID="6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f"
Sep 18 19:33:44.518658 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:33:44.518641    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f\": container with ID starting with 6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f not found: ID does not exist" containerID="6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f"
Sep 18 19:33:44.518694 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.518669    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f"} err="failed to get container status \"6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f\": rpc error: code = NotFound desc = could not find container \"6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f\": container with ID starting with 6715805880fecaf6e91945304dd56892584dbd296b68c6228b3dd7f456aab96f not found: ID does not exist"
Sep 18 19:33:44.518694 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.518687    2559 scope.go:117] "RemoveContainer" containerID="e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7"
Sep 18 19:33:44.518936 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:33:44.518919    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7\": container with ID starting with e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7 not found: ID does not exist" containerID="e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7"
Sep 18 19:33:44.518977 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.518943    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7"} err="failed to get container status \"e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7\": rpc error: code = NotFound desc = could not find container \"e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7\": container with ID starting with e5c5dd0d96b9629d4bc10fa7bfb94cfa290b78a6981f4017b9c6dbdc895555b7 not found: ID does not exist"
Sep 18 19:33:44.518977 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.518958    2559 scope.go:117] "RemoveContainer" containerID="6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9"
Sep 18 19:33:44.519168 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:33:44.519152    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9\": container with ID starting with 6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9 not found: ID does not exist" containerID="6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9"
Sep 18 19:33:44.519258 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.519171    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9"} err="failed to get container status \"6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9\": rpc error: code = NotFound desc = could not find container \"6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9\": container with ID starting with 6576dd9844936064f586ff7354d4200b9f66058cc7c0e09b1e46602644c288e9 not found: ID does not exist"
Sep 18 19:33:44.521537 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:44.521520    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-sfbgw"]
Sep 18 19:33:45.710836 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:45.710805    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" path="/var/lib/kubelet/pods/35b93c78-9094-4b1c-8e40-0dfb10580d71/volumes"
Sep 18 19:33:49.220217 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:49.220171    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:33:50.226086 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:50.226057    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:33:58.705419 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:33:58.705334    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:34:26.524538 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:26.524502    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"]
Sep 18 19:34:26.525117 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:26.524750    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" containerID="cri-o://10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac" gracePeriod=30
Sep 18 19:34:26.525117 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:26.524804    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kube-rbac-proxy" containerID="cri-o://a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c" gracePeriod=30
Sep 18 19:34:26.608559 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:26.608528    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"]
Sep 18 19:34:26.608806 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:26.608767    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" containerID="cri-o://b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a" gracePeriod=30
Sep 18 19:34:26.608870 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:26.608810    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kube-rbac-proxy" containerID="cri-o://74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742" gracePeriod=30
Sep 18 19:34:26.674323 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:26.674290    2559 generic.go:356] "Generic (PLEG): container finished" podID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerID="a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c" exitCode=2
Sep 18 19:34:26.674504 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:26.674321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" event={"ID":"0be91c6e-819c-4c2a-a7c1-9eb7085da62c","Type":"ContainerDied","Data":"a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c"}
Sep 18 19:34:27.681063 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:27.681032    2559 generic.go:356] "Generic (PLEG): container finished" podID="707c46b7-8078-4090-9148-53056998e846" containerID="74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742" exitCode=2
Sep 18 19:34:27.681452 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:27.681083    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" event={"ID":"707c46b7-8078-4090-9148-53056998e846","Type":"ContainerDied","Data":"74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742"}
Sep 18 19:34:28.706112 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:28.706074    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:34:29.214705 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:29.214664    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 18 19:34:30.219803 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.219758    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 19:34:30.225196 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.225169    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:34:30.550980 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.550955    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:34:30.606228 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.606193    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-cabundle-cert\") pod \"707c46b7-8078-4090-9148-53056998e846\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") "
Sep 18 19:34:30.606369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.606238    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-kube-rbac-proxy-sar-config\") pod \"707c46b7-8078-4090-9148-53056998e846\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") "
Sep 18 19:34:30.606369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.606292    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tchq2\" (UniqueName: \"kubernetes.io/projected/707c46b7-8078-4090-9148-53056998e846-kube-api-access-tchq2\") pod \"707c46b7-8078-4090-9148-53056998e846\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") "
Sep 18 19:34:30.606369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.606334    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707c46b7-8078-4090-9148-53056998e846-kserve-provision-location\") pod \"707c46b7-8078-4090-9148-53056998e846\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") "
Sep 18 19:34:30.606523 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.606368    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707c46b7-8078-4090-9148-53056998e846-proxy-tls\") pod \"707c46b7-8078-4090-9148-53056998e846\" (UID: \"707c46b7-8078-4090-9148-53056998e846\") "
Sep 18 19:34:30.606607 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.606585    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "707c46b7-8078-4090-9148-53056998e846" (UID: "707c46b7-8078-4090-9148-53056998e846"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:34:30.606690 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.606662    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "707c46b7-8078-4090-9148-53056998e846" (UID: "707c46b7-8078-4090-9148-53056998e846"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:34:30.606750 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.606723    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:34:30.606800 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.606755    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707c46b7-8078-4090-9148-53056998e846-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "707c46b7-8078-4090-9148-53056998e846" (UID: "707c46b7-8078-4090-9148-53056998e846"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:34:30.608466 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.608433    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/707c46b7-8078-4090-9148-53056998e846-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "707c46b7-8078-4090-9148-53056998e846" (UID: "707c46b7-8078-4090-9148-53056998e846"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:34:30.608544 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.608503    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707c46b7-8078-4090-9148-53056998e846-kube-api-access-tchq2" (OuterVolumeSpecName: "kube-api-access-tchq2") pod "707c46b7-8078-4090-9148-53056998e846" (UID: "707c46b7-8078-4090-9148-53056998e846"). InnerVolumeSpecName "kube-api-access-tchq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:34:30.696037 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.696011    2559 generic.go:356] "Generic (PLEG): container finished" podID="707c46b7-8078-4090-9148-53056998e846" containerID="b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a" exitCode=0
Sep 18 19:34:30.696194 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.696065    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"
Sep 18 19:34:30.696194 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.696073    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" event={"ID":"707c46b7-8078-4090-9148-53056998e846","Type":"ContainerDied","Data":"b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a"}
Sep 18 19:34:30.696194 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.696146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv" event={"ID":"707c46b7-8078-4090-9148-53056998e846","Type":"ContainerDied","Data":"c9abca681ce7df0770eb3a003793c391dbd8efaeebf9d3904e3ba3034d71317b"}
Sep 18 19:34:30.696194 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.696173    2559 scope.go:117] "RemoveContainer" containerID="74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742"
Sep 18 19:34:30.705265 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.705249    2559 scope.go:117] "RemoveContainer" containerID="b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a"
Sep 18 19:34:30.707654 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.707628    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707c46b7-8078-4090-9148-53056998e846-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:34:30.707744 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.707660    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707c46b7-8078-4090-9148-53056998e846-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:34:30.707744 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.707678    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707c46b7-8078-4090-9148-53056998e846-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:34:30.707744 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.707691    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tchq2\" (UniqueName: \"kubernetes.io/projected/707c46b7-8078-4090-9148-53056998e846-kube-api-access-tchq2\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:34:30.713005 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.712991    2559 scope.go:117] "RemoveContainer" containerID="ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1"
Sep 18 19:34:30.720517 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.720502    2559 scope.go:117] "RemoveContainer" containerID="74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742"
Sep 18 19:34:30.720804 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:34:30.720784    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742\": container with ID starting with 74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742 not found: ID does not exist" containerID="74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742"
Sep 18 19:34:30.720853 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.720813    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742"} err="failed to get container status \"74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742\": rpc error: code = NotFound desc = could not find container \"74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742\": container with ID starting with 74ab0c7d79ea3fcdb30cba7b78606a9d863a5872389aa4e85f6fd7dcf6446742 not found: ID does not exist"
Sep 18 19:34:30.720853 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.720830    2559 scope.go:117] "RemoveContainer" containerID="b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a"
Sep 18 19:34:30.720933 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.720867    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"]
Sep 18 19:34:30.721069 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:34:30.721052    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a\": container with ID starting with b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a not found: ID does not exist" containerID="b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a"
Sep 18 19:34:30.721107 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.721072    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a"} err="failed to get container status \"b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a\": rpc error: code = NotFound desc = could not find container \"b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a\": container with ID starting with b4e0ca9177f80ff5e7040a041011f6e0d3e58b50b356c4ccedd6734525e7720a not found: ID does not exist"
Sep 18 19:34:30.721107 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.721085    2559 scope.go:117] "RemoveContainer" containerID="ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1"
Sep 18 19:34:30.721401 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:34:30.721383    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1\": container with ID starting with ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1 not found: ID does not exist" containerID="ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1"
Sep 18 19:34:30.721460 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.721404    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1"} err="failed to get container status \"ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1\": rpc error: code = NotFound desc = could not find container \"ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1\": container with ID starting with ba9947352e29b9b464d240f49e66d53e7e5e869439563289f761fbb4185525a1 not found: ID does not exist"
Sep 18 19:34:30.729620 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:30.729578    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-db55d-predictor-5fbf77b79d-vkqlv"]
Sep 18 19:34:31.475444 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.475422    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:34:31.523034 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.523008    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-rbac-proxy-sar-config\") pod \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") "
Sep 18 19:34:31.523180 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.523064    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kserve-provision-location\") pod \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") "
Sep 18 19:34:31.523180 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.523103    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-proxy-tls\") pod \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") "
Sep 18 19:34:31.523180 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.523168    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq96t\" (UniqueName: \"kubernetes.io/projected/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-api-access-sq96t\") pod \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") "
Sep 18 19:34:31.523369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.523196    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-cabundle-cert\") pod \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\" (UID: \"0be91c6e-819c-4c2a-a7c1-9eb7085da62c\") "
Sep 18 19:34:31.523430 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.523402    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0be91c6e-819c-4c2a-a7c1-9eb7085da62c" (UID: "0be91c6e-819c-4c2a-a7c1-9eb7085da62c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:34:31.523526 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.523434    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0be91c6e-819c-4c2a-a7c1-9eb7085da62c" (UID: "0be91c6e-819c-4c2a-a7c1-9eb7085da62c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:34:31.523589 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.523576    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:34:31.523589 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.523577    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0be91c6e-819c-4c2a-a7c1-9eb7085da62c" (UID: "0be91c6e-819c-4c2a-a7c1-9eb7085da62c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:34:31.523674 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.523591    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:34:31.525304 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.525277    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0be91c6e-819c-4c2a-a7c1-9eb7085da62c" (UID: "0be91c6e-819c-4c2a-a7c1-9eb7085da62c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:34:31.525401 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.525384    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-api-access-sq96t" (OuterVolumeSpecName: "kube-api-access-sq96t") pod "0be91c6e-819c-4c2a-a7c1-9eb7085da62c" (UID: "0be91c6e-819c-4c2a-a7c1-9eb7085da62c"). InnerVolumeSpecName "kube-api-access-sq96t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:34:31.624114 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.624085    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:34:31.624114 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.624109    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq96t\" (UniqueName: \"kubernetes.io/projected/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-kube-api-access-sq96t\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:34:31.624114 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.624119    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be91c6e-819c-4c2a-a7c1-9eb7085da62c-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:34:31.702129 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.702101    2559 generic.go:356] "Generic (PLEG): container finished" podID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerID="10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac" exitCode=0
Sep 18 19:34:31.702288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.702163    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"
Sep 18 19:34:31.702288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.702186    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" event={"ID":"0be91c6e-819c-4c2a-a7c1-9eb7085da62c","Type":"ContainerDied","Data":"10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac"}
Sep 18 19:34:31.702418 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.702290    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd" event={"ID":"0be91c6e-819c-4c2a-a7c1-9eb7085da62c","Type":"ContainerDied","Data":"521e2e22c804de6f1b4483420fe33cd857859605c965446d9e8ffc3abf16097e"}
Sep 18 19:34:31.702418 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.702331    2559 scope.go:117] "RemoveContainer" containerID="a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c"
Sep 18 19:34:31.711110 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.711089    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="707c46b7-8078-4090-9148-53056998e846" path="/var/lib/kubelet/pods/707c46b7-8078-4090-9148-53056998e846/volumes"
Sep 18 19:34:31.711189 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.711107    2559 scope.go:117] "RemoveContainer" containerID="10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac"
Sep 18 19:34:31.718491 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.718477    2559 scope.go:117] "RemoveContainer" containerID="a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e"
Sep 18 19:34:31.725334 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.725315    2559 scope.go:117] "RemoveContainer" containerID="a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c"
Sep 18 19:34:31.725655 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:34:31.725629    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c\": container with ID starting with a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c not found: ID does not exist" containerID="a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c"
Sep 18 19:34:31.725747 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.725659    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c"} err="failed to get container status \"a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c\": rpc error: code = NotFound desc = could not find container \"a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c\": container with ID starting with a2e38f7019e39a65a27f7f70998a6a71abe892ce2522056270585b3ca62b900c not found: ID does not exist"
Sep 18 19:34:31.725747 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.725680    2559 scope.go:117] "RemoveContainer" containerID="10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac"
Sep 18 19:34:31.726016 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:34:31.725987    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac\": container with ID starting with 10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac not found: ID does not exist" containerID="10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac"
Sep 18 19:34:31.726121 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.726024    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac"} err="failed to get container status \"10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac\": rpc error: code = NotFound desc = could not find container \"10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac\": container with ID starting with 10d94d15f5eef6f261de1f4b61882666c0078e87bf8f9ad9b77e9191b066dbac not found: ID does not exist"
Sep 18 19:34:31.726121 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.726044    2559 scope.go:117] "RemoveContainer" containerID="a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e"
Sep 18 19:34:31.726379 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:34:31.726354    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e\": container with ID starting with a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e not found: ID does not exist" containerID="a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e"
Sep 18 19:34:31.726433 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.726385    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e"} err="failed to get container status \"a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e\": rpc error: code = NotFound desc = could not find container \"a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e\": container with ID starting with a4cbde8f7a88a0655eb887f02d938eef1e1932485fe6d6512e513211ae01e39e not found: ID does not exist"
Sep 18 19:34:31.727977 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.727938    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"]
Sep 18 19:34:31.729940 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:31.729921    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-db55d-predictor-556c7fd694-hzpzd"]
Sep 18 19:34:33.710256 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:33.710223    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" path="/var/lib/kubelet/pods/0be91c6e-819c-4c2a-a7c1-9eb7085da62c/volumes"
Sep 18 19:34:39.689693 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:39.689664    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:34:39.690124 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:34:39.689768    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:35:41.663783 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.663748    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"]
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664167    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="storage-initializer"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664186    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="storage-initializer"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664213    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664221    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664235    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kube-rbac-proxy"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664244    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kube-rbac-proxy"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664266    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664277    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664296    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="storage-initializer"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664305    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="storage-initializer"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664314    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kube-rbac-proxy"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664322    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kube-rbac-proxy"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664330    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664335    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664344    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="storage-initializer"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664349    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="storage-initializer"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664357    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kube-rbac-proxy"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664365    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kube-rbac-proxy"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664374    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kube-rbac-proxy"
Sep 18 19:35:41.664369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664382    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kube-rbac-proxy"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664393    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="storage-initializer"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664402    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="storage-initializer"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664411    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664419    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664507    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kserve-container"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664518    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kube-rbac-proxy"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664530    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kube-rbac-proxy"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664541    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kube-rbac-proxy"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664549    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="707c46b7-8078-4090-9148-53056998e846" containerName="kserve-container"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664560    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb55976-52e2-417e-be5e-eaec9bb51fc3" containerName="kserve-container"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664569    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be91c6e-819c-4c2a-a7c1-9eb7085da62c" containerName="kserve-container"
Sep 18 19:35:41.665071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.664578    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b93c78-9094-4b1c-8e40-0dfb10580d71" containerName="kube-rbac-proxy"
Sep 18 19:35:41.667590 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.667574    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.670526 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.670504    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-2a1456-kube-rbac-proxy-sar-config\""
Sep 18 19:35:41.670526 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.670513    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:35:41.670698 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.670536    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-2a1456\""
Sep 18 19:35:41.670698 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.670517    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:35:41.670796 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.670722    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-2a1456-dockercfg-zsh9l\""
Sep 18 19:35:41.670860 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.670802    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-2a1456-predictor-serving-cert\""
Sep 18 19:35:41.671977 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.671958    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:35:41.680675 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.680653    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"]
Sep 18 19:35:41.801439 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.801409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.801439 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.801439    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-cabundle-cert\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.801686 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.801473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-proxy-tls\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.801686 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.801568    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hbtn\" (UniqueName: \"kubernetes.io/projected/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-api-access-5hbtn\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.801686 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.801677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kserve-provision-location\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.902982 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.902953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.903185 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.902991    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-cabundle-cert\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.903185 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.903039    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-proxy-tls\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.903185 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.903079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hbtn\" (UniqueName: \"kubernetes.io/projected/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-api-access-5hbtn\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.903185 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.903144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kserve-provision-location\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.903549 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.903527    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kserve-provision-location\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.903699 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.903680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-cabundle-cert\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.903742 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.903698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.905424 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.905406    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-proxy-tls\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.912056 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.912035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hbtn\" (UniqueName: \"kubernetes.io/projected/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-api-access-5hbtn\") pod \"isvc-secondary-2a1456-predictor-666d6f9868-hnq9j\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") " pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:41.978771 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:41.978700    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:42.251914 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:42.251886    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"]
Sep 18 19:35:42.253005 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:35:42.252979    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc97a40_0936_4f4f_8a58_25d50fe8c5c8.slice/crio-3e879b4442d3acb9cd7dd0546d0005015f003a29d2ec6a5345c55fa69e9e6fcd WatchSource:0}: Error finding container 3e879b4442d3acb9cd7dd0546d0005015f003a29d2ec6a5345c55fa69e9e6fcd: Status 404 returned error can't find the container with id 3e879b4442d3acb9cd7dd0546d0005015f003a29d2ec6a5345c55fa69e9e6fcd
Sep 18 19:35:43.005084 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:43.005044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j" event={"ID":"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8","Type":"ContainerStarted","Data":"d615b249e34503696fb685b7cdac4884337db72498ec98ed2b1c10355e5f6172"}
Sep 18 19:35:43.007382 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:43.005115    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j" event={"ID":"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8","Type":"ContainerStarted","Data":"3e879b4442d3acb9cd7dd0546d0005015f003a29d2ec6a5345c55fa69e9e6fcd"}
Sep 18 19:35:47.021743 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:47.021716    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a1456-predictor-666d6f9868-hnq9j_7dc97a40-0936-4f4f-8a58-25d50fe8c5c8/storage-initializer/0.log"
Sep 18 19:35:47.022190 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:47.021756    2559 generic.go:356] "Generic (PLEG): container finished" podID="7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" containerID="d615b249e34503696fb685b7cdac4884337db72498ec98ed2b1c10355e5f6172" exitCode=1
Sep 18 19:35:47.022190 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:47.021814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j" event={"ID":"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8","Type":"ContainerDied","Data":"d615b249e34503696fb685b7cdac4884337db72498ec98ed2b1c10355e5f6172"}
Sep 18 19:35:48.028090 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:48.028064    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a1456-predictor-666d6f9868-hnq9j_7dc97a40-0936-4f4f-8a58-25d50fe8c5c8/storage-initializer/0.log"
Sep 18 19:35:48.028488 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:48.028149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j" event={"ID":"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8","Type":"ContainerStarted","Data":"ba1a3b81dd99b99279436f9884982082c40a3833c3490f3c57e64587a50ff640"}
Sep 18 19:35:54.053820 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:54.053794    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a1456-predictor-666d6f9868-hnq9j_7dc97a40-0936-4f4f-8a58-25d50fe8c5c8/storage-initializer/1.log"
Sep 18 19:35:54.054222 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:54.054134    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a1456-predictor-666d6f9868-hnq9j_7dc97a40-0936-4f4f-8a58-25d50fe8c5c8/storage-initializer/0.log"
Sep 18 19:35:54.054222 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:54.054161    2559 generic.go:356] "Generic (PLEG): container finished" podID="7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" containerID="ba1a3b81dd99b99279436f9884982082c40a3833c3490f3c57e64587a50ff640" exitCode=1
Sep 18 19:35:54.054331 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:54.054241    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j" event={"ID":"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8","Type":"ContainerDied","Data":"ba1a3b81dd99b99279436f9884982082c40a3833c3490f3c57e64587a50ff640"}
Sep 18 19:35:54.054331 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:54.054307    2559 scope.go:117] "RemoveContainer" containerID="d615b249e34503696fb685b7cdac4884337db72498ec98ed2b1c10355e5f6172"
Sep 18 19:35:54.054723 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:54.054696    2559 scope.go:117] "RemoveContainer" containerID="d615b249e34503696fb685b7cdac4884337db72498ec98ed2b1c10355e5f6172"
Sep 18 19:35:54.064943 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:35:54.064920    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-2a1456-predictor-666d6f9868-hnq9j_kserve-ci-e2e-test(7dc97a40-0936-4f4f-8a58-25d50fe8c5c8)\"" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j" podUID="7dc97a40-0936-4f4f-8a58-25d50fe8c5c8"
Sep 18 19:35:54.065176 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:35:54.065152    2559 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-2a1456-predictor-666d6f9868-hnq9j_kserve-ci-e2e-test_7dc97a40-0936-4f4f-8a58-25d50fe8c5c8_0 in pod sandbox 3e879b4442d3acb9cd7dd0546d0005015f003a29d2ec6a5345c55fa69e9e6fcd from index: no such id: 'd615b249e34503696fb685b7cdac4884337db72498ec98ed2b1c10355e5f6172'" containerID="d615b249e34503696fb685b7cdac4884337db72498ec98ed2b1c10355e5f6172"
Sep 18 19:35:54.065245 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:54.065185    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d615b249e34503696fb685b7cdac4884337db72498ec98ed2b1c10355e5f6172"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-2a1456-predictor-666d6f9868-hnq9j_kserve-ci-e2e-test_7dc97a40-0936-4f4f-8a58-25d50fe8c5c8_0 in pod sandbox 3e879b4442d3acb9cd7dd0546d0005015f003a29d2ec6a5345c55fa69e9e6fcd from index: no such id: 'd615b249e34503696fb685b7cdac4884337db72498ec98ed2b1c10355e5f6172'"
Sep 18 19:35:55.059791 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:55.059766    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a1456-predictor-666d6f9868-hnq9j_7dc97a40-0936-4f4f-8a58-25d50fe8c5c8/storage-initializer/1.log"
Sep 18 19:35:57.731997 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:57.731968    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"]
Sep 18 19:35:57.863399 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:57.863380    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a1456-predictor-666d6f9868-hnq9j_7dc97a40-0936-4f4f-8a58-25d50fe8c5c8/storage-initializer/1.log"
Sep 18 19:35:57.865647 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:57.865632    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:58.043306 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.043282    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hbtn\" (UniqueName: \"kubernetes.io/projected/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-api-access-5hbtn\") pod \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") "
Sep 18 19:35:58.043443 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.043314    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kserve-provision-location\") pod \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") "
Sep 18 19:35:58.043443 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.043351    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-proxy-tls\") pod \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") "
Sep 18 19:35:58.043551 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.043533    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-cabundle-cert\") pod \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") "
Sep 18 19:35:58.043588 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.043544    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" (UID: "7dc97a40-0936-4f4f-8a58-25d50fe8c5c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:35:58.043588 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.043571    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-rbac-proxy-sar-config\") pod \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\" (UID: \"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8\") "
Sep 18 19:35:58.043797 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.043780    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" (UID: "7dc97a40-0936-4f4f-8a58-25d50fe8c5c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:35:58.043862 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.043828    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:35:58.043946 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.043928    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" (UID: "7dc97a40-0936-4f4f-8a58-25d50fe8c5c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:35:58.045439 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.045417    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-api-access-5hbtn" (OuterVolumeSpecName: "kube-api-access-5hbtn") pod "7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" (UID: "7dc97a40-0936-4f4f-8a58-25d50fe8c5c8"). InnerVolumeSpecName "kube-api-access-5hbtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:35:58.045536 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.045515    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" (UID: "7dc97a40-0936-4f4f-8a58-25d50fe8c5c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:35:58.073620 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.073600    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a1456-predictor-666d6f9868-hnq9j_7dc97a40-0936-4f4f-8a58-25d50fe8c5c8/storage-initializer/1.log"
Sep 18 19:35:58.073726 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.073682    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"
Sep 18 19:35:58.073768 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.073679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j" event={"ID":"7dc97a40-0936-4f4f-8a58-25d50fe8c5c8","Type":"ContainerDied","Data":"3e879b4442d3acb9cd7dd0546d0005015f003a29d2ec6a5345c55fa69e9e6fcd"}
Sep 18 19:35:58.073829 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.073817    2559 scope.go:117] "RemoveContainer" containerID="ba1a3b81dd99b99279436f9884982082c40a3833c3490f3c57e64587a50ff640"
Sep 18 19:35:58.118438 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.118405    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"]
Sep 18 19:35:58.123508 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.123476    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a1456-predictor-666d6f9868-hnq9j"]
Sep 18 19:35:58.145233 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.145193    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hbtn\" (UniqueName: \"kubernetes.io/projected/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-api-access-5hbtn\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:35:58.145233 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.145230    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:35:58.145368 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.145243    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:35:58.145368 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:58.145251    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:35:59.710275 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:35:59.710245    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" path="/var/lib/kubelet/pods/7dc97a40-0936-4f4f-8a58-25d50fe8c5c8/volumes"
Sep 18 19:36:03.094792 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.094753    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"]
Sep 18 19:36:03.095280 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.095250    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" containerName="storage-initializer"
Sep 18 19:36:03.095280 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.095269    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" containerName="storage-initializer"
Sep 18 19:36:03.095419 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.095292    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" containerName="storage-initializer"
Sep 18 19:36:03.095419 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.095301    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" containerName="storage-initializer"
Sep 18 19:36:03.095419 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.095389    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" containerName="storage-initializer"
Sep 18 19:36:03.095419 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.095403    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc97a40-0936-4f4f-8a58-25d50fe8c5c8" containerName="storage-initializer"
Sep 18 19:36:03.099908 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.099879    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.103311 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.103285    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:36:03.104088 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.104064    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-dc5230-predictor-serving-cert\""
Sep 18 19:36:03.104176 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.104137    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-dc5230-dockercfg-k29hl\""
Sep 18 19:36:03.104263 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.104089    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:36:03.104263 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.104090    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:36:03.104263 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.104142    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-dc5230-kube-rbac-proxy-sar-config\""
Sep 18 19:36:03.104451 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.104088    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-dc5230\""
Sep 18 19:36:03.113377 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.113358    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"]
Sep 18 19:36:03.293111 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.293077    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-cabundle-cert\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.293288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.293124    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kserve-provision-location\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.293288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.293227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-proxy-tls\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.293288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.293256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.293399 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.293323    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64qr8\" (UniqueName: \"kubernetes.io/projected/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-api-access-64qr8\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.394546 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.394470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64qr8\" (UniqueName: \"kubernetes.io/projected/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-api-access-64qr8\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.394546 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.394522    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-cabundle-cert\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.394764 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.394554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kserve-provision-location\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.394764 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.394585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-proxy-tls\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.394764 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.394609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.395102 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.395078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kserve-provision-location\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.395384 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.395365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-cabundle-cert\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.395384 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.395375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.397119 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.397102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-proxy-tls\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.406519 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.406495    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64qr8\" (UniqueName: \"kubernetes.io/projected/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-api-access-64qr8\") pod \"isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") " pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.410396 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.410372    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:03.692269 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:03.692174    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"]
Sep 18 19:36:03.694614 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:36:03.694587    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad4a3a7_d3f7_4e2a_86f2_fa6a6c1a0b9b.slice/crio-b177812c9330660c002f97a30aa2fc4f5be8f7f5ebbdcd7224da631bcc79fa11 WatchSource:0}: Error finding container b177812c9330660c002f97a30aa2fc4f5be8f7f5ebbdcd7224da631bcc79fa11: Status 404 returned error can't find the container with id b177812c9330660c002f97a30aa2fc4f5be8f7f5ebbdcd7224da631bcc79fa11
Sep 18 19:36:04.101727 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:04.101694    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk" event={"ID":"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b","Type":"ContainerStarted","Data":"7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8"}
Sep 18 19:36:04.102191 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:04.101762    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk" event={"ID":"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b","Type":"ContainerStarted","Data":"b177812c9330660c002f97a30aa2fc4f5be8f7f5ebbdcd7224da631bcc79fa11"}
Sep 18 19:36:10.128603 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:10.128574    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk_bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b/storage-initializer/0.log"
Sep 18 19:36:10.129044 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:10.128613    2559 generic.go:356] "Generic (PLEG): container finished" podID="bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" containerID="7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8" exitCode=1
Sep 18 19:36:10.129044 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:10.128698    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk" event={"ID":"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b","Type":"ContainerDied","Data":"7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8"}
Sep 18 19:36:11.135363 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:11.135337    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk_bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b/storage-initializer/0.log"
Sep 18 19:36:11.135728 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:11.135472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk" event={"ID":"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b","Type":"ContainerStarted","Data":"7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423"}
Sep 18 19:36:12.941142 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:12.941113    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"]
Sep 18 19:36:12.941722 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:12.941345    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk" podUID="bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" containerName="storage-initializer" containerID="cri-o://7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423" gracePeriod=30
Sep 18 19:36:14.380394 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.380373    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk_bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b/storage-initializer/1.log"
Sep 18 19:36:14.380763 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.380750    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk_bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b/storage-initializer/0.log"
Sep 18 19:36:14.382990 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.382976    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:14.487343 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.487278    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64qr8\" (UniqueName: \"kubernetes.io/projected/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-api-access-64qr8\") pod \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") "
Sep 18 19:36:14.487469 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.487344    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-proxy-tls\") pod \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") "
Sep 18 19:36:14.487469 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.487382    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-rbac-proxy-sar-config\") pod \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") "
Sep 18 19:36:14.487469 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.487422    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-cabundle-cert\") pod \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") "
Sep 18 19:36:14.487630 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.487610    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kserve-provision-location\") pod \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\" (UID: \"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b\") "
Sep 18 19:36:14.487786 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.487757    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" (UID: "bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:14.487889 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.487833    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" (UID: "bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:14.487889 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.487768    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" (UID: "bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:14.489520 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.489488    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" (UID: "bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:36:14.489611 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.489547    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-api-access-64qr8" (OuterVolumeSpecName: "kube-api-access-64qr8") pod "bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" (UID: "bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b"). InnerVolumeSpecName "kube-api-access-64qr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:14.588386 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.588355    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:36:14.588386 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.588385    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:36:14.588572 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.588397    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:36:14.588572 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.588410    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:36:14.588572 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:14.588422    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64qr8\" (UniqueName: \"kubernetes.io/projected/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b-kube-api-access-64qr8\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:36:15.152729 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.152697    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk_bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b/storage-initializer/1.log"
Sep 18 19:36:15.153102 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.153087    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk_bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b/storage-initializer/0.log"
Sep 18 19:36:15.153154 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.153121    2559 generic.go:356] "Generic (PLEG): container finished" podID="bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" containerID="7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423" exitCode=1
Sep 18 19:36:15.153193 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.153153    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk" event={"ID":"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b","Type":"ContainerDied","Data":"7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423"}
Sep 18 19:36:15.153193 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.153178    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"
Sep 18 19:36:15.153286 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.153197    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk" event={"ID":"bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b","Type":"ContainerDied","Data":"b177812c9330660c002f97a30aa2fc4f5be8f7f5ebbdcd7224da631bcc79fa11"}
Sep 18 19:36:15.153286 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.153244    2559 scope.go:117] "RemoveContainer" containerID="7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423"
Sep 18 19:36:15.161840 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.161825    2559 scope.go:117] "RemoveContainer" containerID="7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8"
Sep 18 19:36:15.169217 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.169179    2559 scope.go:117] "RemoveContainer" containerID="7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423"
Sep 18 19:36:15.169492 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:36:15.169473    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423\": container with ID starting with 7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423 not found: ID does not exist" containerID="7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423"
Sep 18 19:36:15.169543 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.169500    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423"} err="failed to get container status \"7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423\": rpc error: code = NotFound desc = could not find container \"7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423\": container with ID starting with 7e0271ae5016fef9867629a0d676b5523b4757e1b85b84f96b1878e44045d423 not found: ID does not exist"
Sep 18 19:36:15.169543 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.169515    2559 scope.go:117] "RemoveContainer" containerID="7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8"
Sep 18 19:36:15.169793 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:36:15.169770    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8\": container with ID starting with 7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8 not found: ID does not exist" containerID="7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8"
Sep 18 19:36:15.169834 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.169804    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8"} err="failed to get container status \"7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8\": rpc error: code = NotFound desc = could not find container \"7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8\": container with ID starting with 7c827588bcf406d41020ac421edeec396fa9a947892df54e8dc721ecb73371b8 not found: ID does not exist"
Sep 18 19:36:15.200032 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.200006    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"]
Sep 18 19:36:15.206679 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.206653    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-dc5230-predictor-6b8b945c56-ff4nk"]
Sep 18 19:36:15.710497 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:15.710465    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" path="/var/lib/kubelet/pods/bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b/volumes"
Sep 18 19:36:18.180952 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.180923    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"]
Sep 18 19:36:18.181352 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.181251    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" containerName="storage-initializer"
Sep 18 19:36:18.181352 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.181263    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" containerName="storage-initializer"
Sep 18 19:36:18.181352 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.181286    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" containerName="storage-initializer"
Sep 18 19:36:18.181352 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.181291    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" containerName="storage-initializer"
Sep 18 19:36:18.181352 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.181341    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" containerName="storage-initializer"
Sep 18 19:36:18.181352 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.181351    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad4a3a7-d3f7-4e2a-86f2-fa6a6c1a0b9b" containerName="storage-initializer"
Sep 18 19:36:18.185810 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.185794    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.188596 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.188576    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-9b178-predictor-serving-cert\""
Sep 18 19:36:18.188717 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.188682    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ct9wb\""
Sep 18 19:36:18.188910 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.188893    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-9b178-kube-rbac-proxy-sar-config\""
Sep 18 19:36:18.189021 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.189003    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:36:18.189866 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.189823    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:36:18.189976 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.189894    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:36:18.189976 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.189840    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:36:18.197490 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.197467    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"]
Sep 18 19:36:18.217476 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.217453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.217594 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.217502    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-cabundle-cert\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.217594 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.217524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f48102d-5e04-4c04-8df7-e2483822e5ba-proxy-tls\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.217707 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.217606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w79fn\" (UniqueName: \"kubernetes.io/projected/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-api-access-w79fn\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.217707 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.217642    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f48102d-5e04-4c04-8df7-e2483822e5ba-kserve-provision-location\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.318193 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.318163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.318376 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.318227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-cabundle-cert\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.318376 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.318258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f48102d-5e04-4c04-8df7-e2483822e5ba-proxy-tls\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.318376 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.318285    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w79fn\" (UniqueName: \"kubernetes.io/projected/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-api-access-w79fn\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.318376 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.318303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f48102d-5e04-4c04-8df7-e2483822e5ba-kserve-provision-location\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.318745 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.318719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f48102d-5e04-4c04-8df7-e2483822e5ba-kserve-provision-location\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.318943 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.318925    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-cabundle-cert\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.318995 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.318931    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.320595 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.320570    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f48102d-5e04-4c04-8df7-e2483822e5ba-proxy-tls\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.327159 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.327139    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w79fn\" (UniqueName: \"kubernetes.io/projected/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-api-access-w79fn\") pod \"raw-sklearn-9b178-predictor-5788b5574-qlgv6\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") " pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.496758 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.496675    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:18.774047 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:18.774021    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"]
Sep 18 19:36:18.776310 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:36:18.776283    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f48102d_5e04_4c04_8df7_e2483822e5ba.slice/crio-20ed9b6f5c94c94e4bc4834ece01e210b38a2d6e3b72754e5d842d38e7ba12e1 WatchSource:0}: Error finding container 20ed9b6f5c94c94e4bc4834ece01e210b38a2d6e3b72754e5d842d38e7ba12e1: Status 404 returned error can't find the container with id 20ed9b6f5c94c94e4bc4834ece01e210b38a2d6e3b72754e5d842d38e7ba12e1
Sep 18 19:36:19.172391 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:19.172296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" event={"ID":"8f48102d-5e04-4c04-8df7-e2483822e5ba","Type":"ContainerStarted","Data":"653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20"}
Sep 18 19:36:19.172391 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:19.172365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" event={"ID":"8f48102d-5e04-4c04-8df7-e2483822e5ba","Type":"ContainerStarted","Data":"20ed9b6f5c94c94e4bc4834ece01e210b38a2d6e3b72754e5d842d38e7ba12e1"}
Sep 18 19:36:22.719449 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:36:22.719416    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f48102d_5e04_4c04_8df7_e2483822e5ba.slice/crio-conmon-653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 19:36:23.191036 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:23.191001    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerID="653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20" exitCode=0
Sep 18 19:36:23.191231 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:23.191077    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" event={"ID":"8f48102d-5e04-4c04-8df7-e2483822e5ba","Type":"ContainerDied","Data":"653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20"}
Sep 18 19:36:24.197733 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:24.197704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" event={"ID":"8f48102d-5e04-4c04-8df7-e2483822e5ba","Type":"ContainerStarted","Data":"7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8"}
Sep 18 19:36:24.198231 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:24.197772    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" event={"ID":"8f48102d-5e04-4c04-8df7-e2483822e5ba","Type":"ContainerStarted","Data":"9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752"}
Sep 18 19:36:24.198231 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:24.197993    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:24.221475 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:24.221428    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podStartSLOduration=2.383356679 podStartE2EDuration="6.221414132s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.838057453s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:36:24.219493121 +0000 UTC m=+1306.673631311" watchObservedRunningTime="2026-09-18 19:36:24.221414132 +0000 UTC m=+1306.675552310"
Sep 18 19:36:25.201550 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:25.201517    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:25.202646 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:25.202623    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:36:26.205708 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:26.205667    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:36:31.210655 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:31.210626    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:36:31.211263 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:31.211228    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:36:41.211405 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:41.211364    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:36:51.211981 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:36:51.211892    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:37:01.211470 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:01.211429    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:37:11.211400 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:11.211364    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:37:21.212002 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:21.211962    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:37:31.211385 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:31.211186    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:37:36.705272 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:36.705228    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:37:46.706385 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:46.706351    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:37:48.709035 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:48.708996    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"]
Sep 18 19:37:48.709473 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:48.709255    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kube-rbac-proxy" containerID="cri-o://7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8" gracePeriod=30
Sep 18 19:37:48.709473 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:48.709233    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container" containerID="cri-o://9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752" gracePeriod=30
Sep 18 19:37:49.548913 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:49.548879    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerID="7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8" exitCode=2
Sep 18 19:37:49.549082 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:49.548940    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" event={"ID":"8f48102d-5e04-4c04-8df7-e2483822e5ba","Type":"ContainerDied","Data":"7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8"}
Sep 18 19:37:51.206191 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:51.206153    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 19:37:53.548440 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.548415    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:37:53.569219 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.569172    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerID="9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752" exitCode=0
Sep 18 19:37:53.569366 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.569246    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"
Sep 18 19:37:53.569366 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.569236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" event={"ID":"8f48102d-5e04-4c04-8df7-e2483822e5ba","Type":"ContainerDied","Data":"9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752"}
Sep 18 19:37:53.569489 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.569421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6" event={"ID":"8f48102d-5e04-4c04-8df7-e2483822e5ba","Type":"ContainerDied","Data":"20ed9b6f5c94c94e4bc4834ece01e210b38a2d6e3b72754e5d842d38e7ba12e1"}
Sep 18 19:37:53.569489 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.569477    2559 scope.go:117] "RemoveContainer" containerID="7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8"
Sep 18 19:37:53.578644 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.578616    2559 scope.go:117] "RemoveContainer" containerID="9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752"
Sep 18 19:37:53.585930 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.585913    2559 scope.go:117] "RemoveContainer" containerID="653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20"
Sep 18 19:37:53.593723 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.593703    2559 scope.go:117] "RemoveContainer" containerID="7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8"
Sep 18 19:37:53.593979 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:37:53.593949    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8\": container with ID starting with 7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8 not found: ID does not exist" containerID="7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8"
Sep 18 19:37:53.594068 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.593978    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8"} err="failed to get container status \"7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8\": rpc error: code = NotFound desc = could not find container \"7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8\": container with ID starting with 7f98f5c43407e1377bcdd110b847605facf046167fd64e5b709e3d077fc6b8d8 not found: ID does not exist"
Sep 18 19:37:53.594068 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.594002    2559 scope.go:117] "RemoveContainer" containerID="9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752"
Sep 18 19:37:53.594426 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:37:53.594377    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752\": container with ID starting with 9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752 not found: ID does not exist" containerID="9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752"
Sep 18 19:37:53.594426 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.594400    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752"} err="failed to get container status \"9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752\": rpc error: code = NotFound desc = could not find container \"9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752\": container with ID starting with 9ba1f7b4d1b5114fb054da703f2ad63d812eadc289f39dcf54ab47912e6a6752 not found: ID does not exist"
Sep 18 19:37:53.594426 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.594415    2559 scope.go:117] "RemoveContainer" containerID="653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20"
Sep 18 19:37:53.594664 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:37:53.594647    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20\": container with ID starting with 653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20 not found: ID does not exist" containerID="653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20"
Sep 18 19:37:53.594730 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.594668    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20"} err="failed to get container status \"653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20\": rpc error: code = NotFound desc = could not find container \"653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20\": container with ID starting with 653a6d7653b756e8e19398532d104422c39cdf9e2fc4f4740fb3fa050eba9e20 not found: ID does not exist"
Sep 18 19:37:53.643130 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.643100    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f48102d-5e04-4c04-8df7-e2483822e5ba-proxy-tls\") pod \"8f48102d-5e04-4c04-8df7-e2483822e5ba\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") "
Sep 18 19:37:53.643302 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.643138    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w79fn\" (UniqueName: \"kubernetes.io/projected/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-api-access-w79fn\") pod \"8f48102d-5e04-4c04-8df7-e2483822e5ba\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") "
Sep 18 19:37:53.643302 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.643239    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-rbac-proxy-sar-config\") pod \"8f48102d-5e04-4c04-8df7-e2483822e5ba\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") "
Sep 18 19:37:53.643302 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.643273    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-cabundle-cert\") pod \"8f48102d-5e04-4c04-8df7-e2483822e5ba\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") "
Sep 18 19:37:53.643302 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.643290    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f48102d-5e04-4c04-8df7-e2483822e5ba-kserve-provision-location\") pod \"8f48102d-5e04-4c04-8df7-e2483822e5ba\" (UID: \"8f48102d-5e04-4c04-8df7-e2483822e5ba\") "
Sep 18 19:37:53.643659 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.643632    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f48102d-5e04-4c04-8df7-e2483822e5ba" (UID: "8f48102d-5e04-4c04-8df7-e2483822e5ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:37:53.643817 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.643781    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f48102d-5e04-4c04-8df7-e2483822e5ba" (UID: "8f48102d-5e04-4c04-8df7-e2483822e5ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:37:53.643933 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.643881    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f48102d-5e04-4c04-8df7-e2483822e5ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f48102d-5e04-4c04-8df7-e2483822e5ba" (UID: "8f48102d-5e04-4c04-8df7-e2483822e5ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:37:53.645266 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.645245    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f48102d-5e04-4c04-8df7-e2483822e5ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f48102d-5e04-4c04-8df7-e2483822e5ba" (UID: "8f48102d-5e04-4c04-8df7-e2483822e5ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:37:53.645348 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.645337    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-api-access-w79fn" (OuterVolumeSpecName: "kube-api-access-w79fn") pod "8f48102d-5e04-4c04-8df7-e2483822e5ba" (UID: "8f48102d-5e04-4c04-8df7-e2483822e5ba"). InnerVolumeSpecName "kube-api-access-w79fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:37:53.744026 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.743999    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:37:53.744026 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.744029    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f48102d-5e04-4c04-8df7-e2483822e5ba-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:37:53.744216 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.744042    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f48102d-5e04-4c04-8df7-e2483822e5ba-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:37:53.744216 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.744058    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w79fn\" (UniqueName: \"kubernetes.io/projected/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-api-access-w79fn\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:37:53.744216 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.744070    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f48102d-5e04-4c04-8df7-e2483822e5ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:37:53.891991 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.891921    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"]
Sep 18 19:37:53.896809 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:53.896786    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9b178-predictor-5788b5574-qlgv6"]
Sep 18 19:37:55.710698 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:37:55.710661    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" path="/var/lib/kubelet/pods/8f48102d-5e04-4c04-8df7-e2483822e5ba/volumes"
Sep 18 19:38:58.675158 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.675116    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"]
Sep 18 19:38:58.675611 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.675521    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="storage-initializer"
Sep 18 19:38:58.675611 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.675533    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="storage-initializer"
Sep 18 19:38:58.675611 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.675552    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container"
Sep 18 19:38:58.675611 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.675558    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container"
Sep 18 19:38:58.675611 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.675569    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kube-rbac-proxy"
Sep 18 19:38:58.675611 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.675577    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kube-rbac-proxy"
Sep 18 19:38:58.675803 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.675634    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kserve-container"
Sep 18 19:38:58.675803 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.675646    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f48102d-5e04-4c04-8df7-e2483822e5ba" containerName="kube-rbac-proxy"
Sep 18 19:38:58.679975 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.679955    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.682702 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.682682    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:38:58.683907 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.683889    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:38:58.683987 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.683972    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:38:58.685010 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.684989    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 18 19:38:58.685230 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.685020    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ct9wb\""
Sep 18 19:38:58.685338 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.685029    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:38:58.685338 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.685062    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 19:38:58.706376 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.706351    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"]
Sep 18 19:38:58.788817 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.788795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.788970 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.788839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.788970 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.788899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhrnz\" (UniqueName: \"kubernetes.io/projected/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-api-access-fhrnz\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.788970 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.788952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.789144 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.788980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.810050 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.810027    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"]
Sep 18 19:38:58.814271 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.814255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.817454 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.817435    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 19:38:58.831187 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.831165    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"]
Sep 18 19:38:58.890369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.890342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nmc8\" (UniqueName: \"kubernetes.io/projected/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-api-access-9nmc8\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.890525 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.890379    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.890525 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.890408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.890525 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.890460    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.890717 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.890610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhrnz\" (UniqueName: \"kubernetes.io/projected/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-api-access-fhrnz\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.890717 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.890665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.890717 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.890699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a30283-3cf2-491a-87ef-3747ab287ab9-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.890885 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.890728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.890885 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.890787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a30283-3cf2-491a-87ef-3747ab287ab9-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.890885 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.890821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.891043 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:38:58.890937    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 18 19:38:58.891043 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:38:58.891010    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-proxy-tls podName:1ff2e7a2-00ac-4e45-82b4-1016dd452d0b nodeName:}" failed. No retries permitted until 2026-09-18 19:38:59.390990138 +0000 UTC m=+1461.845128308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-proxy-tls") pod "isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" (UID: "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 18 19:38:58.891164 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.891078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.891164 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.891148    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.891378 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.891362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.902618 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.902599    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhrnz\" (UniqueName: \"kubernetes.io/projected/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-api-access-fhrnz\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:58.992124 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.992043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a30283-3cf2-491a-87ef-3747ab287ab9-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.992124 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.992087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a30283-3cf2-491a-87ef-3747ab287ab9-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.992350 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.992126    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nmc8\" (UniqueName: \"kubernetes.io/projected/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-api-access-9nmc8\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.992350 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.992152    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.992350 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.992174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.992706 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.992679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a30283-3cf2-491a-87ef-3747ab287ab9-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.992984 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.992963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.993054 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.992963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:58.995109 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:58.995074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a30283-3cf2-491a-87ef-3747ab287ab9-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:59.001111 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:59.001088    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nmc8\" (UniqueName: \"kubernetes.io/projected/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-api-access-9nmc8\") pod \"isvc-canary-create-predictor-7b568f75-wcgnt\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:59.124141 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:59.124105    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:38:59.395859 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:59.395823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:59.398223 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:59.398186    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:59.407527 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:59.407504    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"]
Sep 18 19:38:59.409186 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:38:59.409163    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a30283_3cf2_491a_87ef_3747ab287ab9.slice/crio-f164d81d3e05b9d0fccdf25b9eb678f425a4c0d5f4a732ce3f7e5acf2557b467 WatchSource:0}: Error finding container f164d81d3e05b9d0fccdf25b9eb678f425a4c0d5f4a732ce3f7e5acf2557b467: Status 404 returned error can't find the container with id f164d81d3e05b9d0fccdf25b9eb678f425a4c0d5f4a732ce3f7e5acf2557b467
Sep 18 19:38:59.590882 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:59.590842    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:38:59.840216 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:59.840160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" event={"ID":"26a30283-3cf2-491a-87ef-3747ab287ab9","Type":"ContainerStarted","Data":"22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b"}
Sep 18 19:38:59.840582 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:59.840242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" event={"ID":"26a30283-3cf2-491a-87ef-3747ab287ab9","Type":"ContainerStarted","Data":"f164d81d3e05b9d0fccdf25b9eb678f425a4c0d5f4a732ce3f7e5acf2557b467"}
Sep 18 19:38:59.881037 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:38:59.881007    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"]
Sep 18 19:38:59.882883 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:38:59.882854    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff2e7a2_00ac_4e45_82b4_1016dd452d0b.slice/crio-0d34db473c13cbee2e4f52cc3c688abaa527bd7ac90a4a5647b6baac2efb24d7 WatchSource:0}: Error finding container 0d34db473c13cbee2e4f52cc3c688abaa527bd7ac90a4a5647b6baac2efb24d7: Status 404 returned error can't find the container with id 0d34db473c13cbee2e4f52cc3c688abaa527bd7ac90a4a5647b6baac2efb24d7
Sep 18 19:39:00.846272 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:00.846231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" event={"ID":"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b","Type":"ContainerStarted","Data":"409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93"}
Sep 18 19:39:00.846649 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:00.846297    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" event={"ID":"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b","Type":"ContainerStarted","Data":"0d34db473c13cbee2e4f52cc3c688abaa527bd7ac90a4a5647b6baac2efb24d7"}
Sep 18 19:39:03.859699 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:03.859664    2559 generic.go:356] "Generic (PLEG): container finished" podID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerID="22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b" exitCode=0
Sep 18 19:39:03.860114 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:03.859709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" event={"ID":"26a30283-3cf2-491a-87ef-3747ab287ab9","Type":"ContainerDied","Data":"22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b"}
Sep 18 19:39:04.865485 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:04.865452    2559 generic.go:356] "Generic (PLEG): container finished" podID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerID="409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93" exitCode=0
Sep 18 19:39:04.865955 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:04.865529    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" event={"ID":"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b","Type":"ContainerDied","Data":"409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93"}
Sep 18 19:39:04.867801 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:04.867779    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" event={"ID":"26a30283-3cf2-491a-87ef-3747ab287ab9","Type":"ContainerStarted","Data":"ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02"}
Sep 18 19:39:04.867901 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:04.867829    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" event={"ID":"26a30283-3cf2-491a-87ef-3747ab287ab9","Type":"ContainerStarted","Data":"4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6"}
Sep 18 19:39:04.870649 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:04.870629    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:39:04.870740 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:04.870678    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:39:04.871942 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:04.871918    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 19:39:05.873742 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:05.873705    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" event={"ID":"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b","Type":"ContainerStarted","Data":"06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc"}
Sep 18 19:39:05.874071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:05.873773    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" event={"ID":"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b","Type":"ContainerStarted","Data":"9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c"}
Sep 18 19:39:05.874071 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:05.873984    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 19:39:05.874233 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:05.874193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:39:05.874343 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:05.874269    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:39:05.875576 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:05.875556    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:39:05.900617 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:05.900574    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podStartSLOduration=3.899809243 podStartE2EDuration="7.900559957s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.000750714s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:05.899078994 +0000 UTC m=+1468.353217168" watchObservedRunningTime="2026-09-18 19:39:05.900559957 +0000 UTC m=+1468.354698134"
Sep 18 19:39:05.900785 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:05.900764    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podStartSLOduration=4.059297305 podStartE2EDuration="7.900756708s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841459403s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:04.933416006 +0000 UTC m=+1467.387554182" watchObservedRunningTime="2026-09-18 19:39:05.900756708 +0000 UTC m=+1468.354894886"
Sep 18 19:39:06.878066 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:06.878025    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:39:10.878735 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:10.878708    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:39:10.879265 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:10.879190    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 19:39:11.883219 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:11.883175    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:39:11.883807 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:11.883781    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:39:20.879616 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:20.879580    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 19:39:21.884413 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:21.884376    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:39:30.879664 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:30.879627    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 19:39:31.884045 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:31.884006    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:39:39.719287 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:39.719250    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:39:39.720649 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:39.720628    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:39:40.879244 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:40.879210    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 19:39:41.883768 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:41.883725    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:39:50.880111 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:50.880075    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 19:39:51.884361 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:39:51.884324    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:40:00.879555 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:00.879510    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 19:40:01.884000 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:01.883954    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:40:10.879514 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:10.879478    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 19:40:11.884703 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:11.884667    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:40:20.879710 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:20.879674    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 19:40:21.883828 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:21.883788    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:40:30.880350 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:30.880323    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:40:31.884932 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:31.884902    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:40:32.048132 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:32.048094    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"]
Sep 18 19:40:32.048387 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:32.048361    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container" containerID="cri-o://4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6" gracePeriod=30
Sep 18 19:40:32.048473 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:32.048422    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kube-rbac-proxy" containerID="cri-o://ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02" gracePeriod=30
Sep 18 19:40:32.237144 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:32.237113    2559 generic.go:356] "Generic (PLEG): container finished" podID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerID="ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02" exitCode=2
Sep 18 19:40:32.237330 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:32.237173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" event={"ID":"26a30283-3cf2-491a-87ef-3747ab287ab9","Type":"ContainerDied","Data":"ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02"}
Sep 18 19:40:35.874382 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:35.874342    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 18 19:40:36.699022 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.698998    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:40:36.803831 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.803738    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a30283-3cf2-491a-87ef-3747ab287ab9-proxy-tls\") pod \"26a30283-3cf2-491a-87ef-3747ab287ab9\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") "
Sep 18 19:40:36.803831 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.803807    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nmc8\" (UniqueName: \"kubernetes.io/projected/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-api-access-9nmc8\") pod \"26a30283-3cf2-491a-87ef-3747ab287ab9\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") "
Sep 18 19:40:36.804053 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.803849    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a30283-3cf2-491a-87ef-3747ab287ab9-kserve-provision-location\") pod \"26a30283-3cf2-491a-87ef-3747ab287ab9\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") "
Sep 18 19:40:36.804053 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.803896    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-cabundle-cert\") pod \"26a30283-3cf2-491a-87ef-3747ab287ab9\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") "
Sep 18 19:40:36.804053 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.803918    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-rbac-proxy-sar-config\") pod \"26a30283-3cf2-491a-87ef-3747ab287ab9\" (UID: \"26a30283-3cf2-491a-87ef-3747ab287ab9\") "
Sep 18 19:40:36.804258 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.804161    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a30283-3cf2-491a-87ef-3747ab287ab9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26a30283-3cf2-491a-87ef-3747ab287ab9" (UID: "26a30283-3cf2-491a-87ef-3747ab287ab9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:40:36.804328 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.804275    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26a30283-3cf2-491a-87ef-3747ab287ab9" (UID: "26a30283-3cf2-491a-87ef-3747ab287ab9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:36.804328 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.804316    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26a30283-3cf2-491a-87ef-3747ab287ab9" (UID: "26a30283-3cf2-491a-87ef-3747ab287ab9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:36.805928 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.805898    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a30283-3cf2-491a-87ef-3747ab287ab9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26a30283-3cf2-491a-87ef-3747ab287ab9" (UID: "26a30283-3cf2-491a-87ef-3747ab287ab9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:36.806028 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.805956    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-api-access-9nmc8" (OuterVolumeSpecName: "kube-api-access-9nmc8") pod "26a30283-3cf2-491a-87ef-3747ab287ab9" (UID: "26a30283-3cf2-491a-87ef-3747ab287ab9"). InnerVolumeSpecName "kube-api-access-9nmc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:36.905219 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.905171    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a30283-3cf2-491a-87ef-3747ab287ab9-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:40:36.905219 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.905197    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nmc8\" (UniqueName: \"kubernetes.io/projected/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-api-access-9nmc8\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:40:36.905612 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.905232    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a30283-3cf2-491a-87ef-3747ab287ab9-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:40:36.905612 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.905241    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:40:36.905612 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:36.905250    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a30283-3cf2-491a-87ef-3747ab287ab9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.260426 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.260338    2559 generic.go:356] "Generic (PLEG): container finished" podID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerID="4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6" exitCode=0
Sep 18 19:40:37.260426 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.260407    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" event={"ID":"26a30283-3cf2-491a-87ef-3747ab287ab9","Type":"ContainerDied","Data":"4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6"}
Sep 18 19:40:37.260600 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.260408    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"
Sep 18 19:40:37.260600 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.260497    2559 scope.go:117] "RemoveContainer" containerID="ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02"
Sep 18 19:40:37.260677 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.260486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt" event={"ID":"26a30283-3cf2-491a-87ef-3747ab287ab9","Type":"ContainerDied","Data":"f164d81d3e05b9d0fccdf25b9eb678f425a4c0d5f4a732ce3f7e5acf2557b467"}
Sep 18 19:40:37.269131 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.269111    2559 scope.go:117] "RemoveContainer" containerID="4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6"
Sep 18 19:40:37.276356 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.276341    2559 scope.go:117] "RemoveContainer" containerID="22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b"
Sep 18 19:40:37.283570 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.283553    2559 scope.go:117] "RemoveContainer" containerID="ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02"
Sep 18 19:40:37.283855 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:40:37.283832    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02\": container with ID starting with ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02 not found: ID does not exist" containerID="ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02"
Sep 18 19:40:37.283909 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.283866    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02"} err="failed to get container status \"ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02\": rpc error: code = NotFound desc = could not find container \"ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02\": container with ID starting with ed6bbb965a31321ff3a4290a29e8e65bd7249d45d5929a5e91e24699c4a89a02 not found: ID does not exist"
Sep 18 19:40:37.283909 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.283890    2559 scope.go:117] "RemoveContainer" containerID="4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6"
Sep 18 19:40:37.284161 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:40:37.284142    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6\": container with ID starting with 4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6 not found: ID does not exist" containerID="4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6"
Sep 18 19:40:37.284226 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.284169    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6"} err="failed to get container status \"4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6\": rpc error: code = NotFound desc = could not find container \"4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6\": container with ID starting with 4a5e48c5c7cc35516f349c21e7f3070eee1a2488699bd7500537c9c3f9a5f8d6 not found: ID does not exist"
Sep 18 19:40:37.284226 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.284184    2559 scope.go:117] "RemoveContainer" containerID="22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b"
Sep 18 19:40:37.284459 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:40:37.284435    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b\": container with ID starting with 22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b not found: ID does not exist" containerID="22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b"
Sep 18 19:40:37.284544 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.284462    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b"} err="failed to get container status \"22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b\": rpc error: code = NotFound desc = could not find container \"22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b\": container with ID starting with 22a66e17a49264d90b8dcfa797e74b82f00deafdd7ee50504f2d3f0a84c2ad3b not found: ID does not exist"
Sep 18 19:40:37.284613 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.284598    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"]
Sep 18 19:40:37.289988 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.289970    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-wcgnt"]
Sep 18 19:40:37.709601 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:37.709573    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" path="/var/lib/kubelet/pods/26a30283-3cf2-491a-87ef-3747ab287ab9/volumes"
Sep 18 19:40:38.791691 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:38.791660    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"]
Sep 18 19:40:38.792258 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:38.791957    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" containerID="cri-o://9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c" gracePeriod=30
Sep 18 19:40:38.792258 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:38.792032    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kube-rbac-proxy" containerID="cri-o://06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc" gracePeriod=30
Sep 18 19:40:39.272238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:39.272185    2559 generic.go:356] "Generic (PLEG): container finished" podID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerID="06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc" exitCode=2
Sep 18 19:40:39.272394 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:39.272235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" event={"ID":"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b","Type":"ContainerDied","Data":"06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc"}
Sep 18 19:40:41.879150 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:41.879105    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 18 19:40:41.884421 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:41.884392    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:40:43.536507 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.536481    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:40:43.662345 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.662265    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-proxy-tls\") pod \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") "
Sep 18 19:40:43.662345 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.662313    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kserve-provision-location\") pod \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") "
Sep 18 19:40:43.662345 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.662334    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-cabundle-cert\") pod \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") "
Sep 18 19:40:43.662639 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.662380    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhrnz\" (UniqueName: \"kubernetes.io/projected/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-api-access-fhrnz\") pod \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") "
Sep 18 19:40:43.662639 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.662408    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-rbac-proxy-sar-config\") pod \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\" (UID: \"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b\") "
Sep 18 19:40:43.662763 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.662661    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" (UID: "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:40:43.662824 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.662699    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" (UID: "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:43.662864 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.662819    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" (UID: "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:43.664528 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.664501    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-api-access-fhrnz" (OuterVolumeSpecName: "kube-api-access-fhrnz") pod "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" (UID: "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b"). InnerVolumeSpecName "kube-api-access-fhrnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:43.664624 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.664576    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" (UID: "1ff2e7a2-00ac-4e45-82b4-1016dd452d0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:43.763560 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.763530    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:40:43.763560 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.763558    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:40:43.763560 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.763567    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:40:43.763759 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.763577    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhrnz\" (UniqueName: \"kubernetes.io/projected/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-api-access-fhrnz\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:40:43.763759 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:43.763585    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:40:44.088445 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088410    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"]
Sep 18 19:40:44.088735 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088724    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="storage-initializer"
Sep 18 19:40:44.088777 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088738    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="storage-initializer"
Sep 18 19:40:44.088777 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088746    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kube-rbac-proxy"
Sep 18 19:40:44.088777 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088751    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kube-rbac-proxy"
Sep 18 19:40:44.088777 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088758    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kube-rbac-proxy"
Sep 18 19:40:44.088777 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088764    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kube-rbac-proxy"
Sep 18 19:40:44.088777 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088776    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="storage-initializer"
Sep 18 19:40:44.088959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088781    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="storage-initializer"
Sep 18 19:40:44.088959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088791    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container"
Sep 18 19:40:44.088959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088796    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container"
Sep 18 19:40:44.088959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088808    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container"
Sep 18 19:40:44.088959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088813    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container"
Sep 18 19:40:44.088959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088859    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kube-rbac-proxy"
Sep 18 19:40:44.088959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088868    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kube-rbac-proxy"
Sep 18 19:40:44.088959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088875    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a30283-3cf2-491a-87ef-3747ab287ab9" containerName="kserve-container"
Sep 18 19:40:44.088959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.088882    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerName="kserve-container"
Sep 18 19:40:44.092010 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.091993    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.094712 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.094692    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 19:40:44.094832 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.094730    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 19:40:44.107806 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.107785    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"]
Sep 18 19:40:44.166416 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.166386    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.166610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.166454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c05d354-a20f-4d74-957a-2e6323f0c241-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.166610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.166474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c05d354-a20f-4d74-957a-2e6323f0c241-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.166610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.166497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.166610 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.166522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7l9x\" (UniqueName: \"kubernetes.io/projected/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-api-access-x7l9x\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.267997 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.267965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c05d354-a20f-4d74-957a-2e6323f0c241-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.267997 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.267997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c05d354-a20f-4d74-957a-2e6323f0c241-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.268261 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.268031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.268261 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.268060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7l9x\" (UniqueName: \"kubernetes.io/projected/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-api-access-x7l9x\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.268261 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.268105    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.268595 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.268563    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c05d354-a20f-4d74-957a-2e6323f0c241-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.268817 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.268799    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.268867 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.268826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.270434 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.270418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c05d354-a20f-4d74-957a-2e6323f0c241-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.276524 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.276506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7l9x\" (UniqueName: \"kubernetes.io/projected/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-api-access-x7l9x\") pod \"isvc-canary-promote-predictor-75f64ff6c9-xczvl\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.298174 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.298147    2559 generic.go:356] "Generic (PLEG): container finished" podID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" containerID="9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c" exitCode=0
Sep 18 19:40:44.298295 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.298225    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"
Sep 18 19:40:44.298295 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.298243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" event={"ID":"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b","Type":"ContainerDied","Data":"9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c"}
Sep 18 19:40:44.298384 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.298310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v" event={"ID":"1ff2e7a2-00ac-4e45-82b4-1016dd452d0b","Type":"ContainerDied","Data":"0d34db473c13cbee2e4f52cc3c688abaa527bd7ac90a4a5647b6baac2efb24d7"}
Sep 18 19:40:44.298384 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.298349    2559 scope.go:117] "RemoveContainer" containerID="06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc"
Sep 18 19:40:44.306384 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.306362    2559 scope.go:117] "RemoveContainer" containerID="9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c"
Sep 18 19:40:44.313486 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.313467    2559 scope.go:117] "RemoveContainer" containerID="409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93"
Sep 18 19:40:44.318796 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.318774    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"]
Sep 18 19:40:44.321360 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.321343    2559 scope.go:117] "RemoveContainer" containerID="06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc"
Sep 18 19:40:44.321639 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:40:44.321622    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc\": container with ID starting with 06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc not found: ID does not exist" containerID="06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc"
Sep 18 19:40:44.321707 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.321651    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc"} err="failed to get container status \"06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc\": rpc error: code = NotFound desc = could not find container \"06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc\": container with ID starting with 06cfd6c71f98fa9bb074590d22d6bc301eb6e5a1cb530d265e31c96ab08d01fc not found: ID does not exist"
Sep 18 19:40:44.321707 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.321676    2559 scope.go:117] "RemoveContainer" containerID="9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c"
Sep 18 19:40:44.321958 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:40:44.321942    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c\": container with ID starting with 9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c not found: ID does not exist" containerID="9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c"
Sep 18 19:40:44.322021 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.321965    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c"} err="failed to get container status \"9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c\": rpc error: code = NotFound desc = could not find container \"9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c\": container with ID starting with 9dcf1b9aaf0984ce1a2c11b807be8b5ca90c41baff7d6e03ba476bfea1968f8c not found: ID does not exist"
Sep 18 19:40:44.322021 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.321983    2559 scope.go:117] "RemoveContainer" containerID="409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93"
Sep 18 19:40:44.322260 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:40:44.322242    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93\": container with ID starting with 409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93 not found: ID does not exist" containerID="409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93"
Sep 18 19:40:44.322299 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.322264    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93"} err="failed to get container status \"409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93\": rpc error: code = NotFound desc = could not find container \"409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93\": container with ID starting with 409b016a5e8d6f4525e3bb575fcc94cd328bf79ef3b6356229a52b1a1b242b93 not found: ID does not exist"
Sep 18 19:40:44.322423 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.322406    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-jcz2v"]
Sep 18 19:40:44.402442 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.402368    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:44.695277 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:44.695252    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"]
Sep 18 19:40:44.697785 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:40:44.697759    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c05d354_a20f_4d74_957a_2e6323f0c241.slice/crio-7dbaffeb10e85e231b00e4988551a9c1a52d11171fe3689e1e1d7451331ef930 WatchSource:0}: Error finding container 7dbaffeb10e85e231b00e4988551a9c1a52d11171fe3689e1e1d7451331ef930: Status 404 returned error can't find the container with id 7dbaffeb10e85e231b00e4988551a9c1a52d11171fe3689e1e1d7451331ef930
Sep 18 19:40:45.306497 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:45.306462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" event={"ID":"0c05d354-a20f-4d74-957a-2e6323f0c241","Type":"ContainerStarted","Data":"6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c"}
Sep 18 19:40:45.306683 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:45.306538    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" event={"ID":"0c05d354-a20f-4d74-957a-2e6323f0c241","Type":"ContainerStarted","Data":"7dbaffeb10e85e231b00e4988551a9c1a52d11171fe3689e1e1d7451331ef930"}
Sep 18 19:40:45.714979 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:45.714902    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ff2e7a2-00ac-4e45-82b4-1016dd452d0b" path="/var/lib/kubelet/pods/1ff2e7a2-00ac-4e45-82b4-1016dd452d0b/volumes"
Sep 18 19:40:49.323298 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:49.323256    2559 generic.go:356] "Generic (PLEG): container finished" podID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerID="6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c" exitCode=0
Sep 18 19:40:49.323703 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:49.323332    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" event={"ID":"0c05d354-a20f-4d74-957a-2e6323f0c241","Type":"ContainerDied","Data":"6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c"}
Sep 18 19:40:50.329172 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:50.329139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" event={"ID":"0c05d354-a20f-4d74-957a-2e6323f0c241","Type":"ContainerStarted","Data":"ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7"}
Sep 18 19:40:50.329558 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:50.329220    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" event={"ID":"0c05d354-a20f-4d74-957a-2e6323f0c241","Type":"ContainerStarted","Data":"3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc"}
Sep 18 19:40:50.329558 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:50.329419    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:50.352169 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:50.352123    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podStartSLOduration=2.400010571 podStartE2EDuration="6.352109085s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.952098514s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:50.350783448 +0000 UTC m=+1572.804921635" watchObservedRunningTime="2026-09-18 19:40:50.352109085 +0000 UTC m=+1572.806247262"
Sep 18 19:40:51.333968 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:51.333937    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:51.335064 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:51.335036    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:40:52.338078 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:52.338030    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:40:57.343403 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:57.343367    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:40:57.344013 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:40:57.343980    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:41:07.344434 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:41:07.344394    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:41:17.344709 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:41:17.344664    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:41:27.344520 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:41:27.344479    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:41:37.344371 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:41:37.344330    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:41:47.343983 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:41:47.343944    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:41:57.344278 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:41:57.344229    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:42:00.706125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:00.706078    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:42:05.805513 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:05.805475    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"]
Sep 18 19:42:05.805959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:05.805740    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" containerID="cri-o://3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc" gracePeriod=30
Sep 18 19:42:05.805959 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:05.805795    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kube-rbac-proxy" containerID="cri-o://ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7" gracePeriod=30
Sep 18 19:42:05.806099 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:05.806020    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 19:42:06.648826 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:06.648793    2559 generic.go:356] "Generic (PLEG): container finished" podID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerID="ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7" exitCode=2
Sep 18 19:42:06.648993 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:06.648865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" event={"ID":"0c05d354-a20f-4d74-957a-2e6323f0c241","Type":"ContainerDied","Data":"ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7"}
Sep 18 19:42:07.339321 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:07.339268    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 18 19:42:10.847183 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.847160    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:42:10.916640 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.916609    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-rbac-proxy-sar-config\") pod \"0c05d354-a20f-4d74-957a-2e6323f0c241\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") "
Sep 18 19:42:10.916784 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.916662    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c05d354-a20f-4d74-957a-2e6323f0c241-proxy-tls\") pod \"0c05d354-a20f-4d74-957a-2e6323f0c241\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") "
Sep 18 19:42:10.916784 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.916683    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7l9x\" (UniqueName: \"kubernetes.io/projected/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-api-access-x7l9x\") pod \"0c05d354-a20f-4d74-957a-2e6323f0c241\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") "
Sep 18 19:42:10.916784 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.916702    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-cabundle-cert\") pod \"0c05d354-a20f-4d74-957a-2e6323f0c241\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") "
Sep 18 19:42:10.916784 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.916751    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c05d354-a20f-4d74-957a-2e6323f0c241-kserve-provision-location\") pod \"0c05d354-a20f-4d74-957a-2e6323f0c241\" (UID: \"0c05d354-a20f-4d74-957a-2e6323f0c241\") "
Sep 18 19:42:10.917101 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.917078    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c05d354-a20f-4d74-957a-2e6323f0c241" (UID: "0c05d354-a20f-4d74-957a-2e6323f0c241"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:42:10.917163 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.917094    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c05d354-a20f-4d74-957a-2e6323f0c241-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c05d354-a20f-4d74-957a-2e6323f0c241" (UID: "0c05d354-a20f-4d74-957a-2e6323f0c241"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:42:10.917163 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.917103    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c05d354-a20f-4d74-957a-2e6323f0c241" (UID: "0c05d354-a20f-4d74-957a-2e6323f0c241"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:42:10.918748 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.918725    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c05d354-a20f-4d74-957a-2e6323f0c241-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c05d354-a20f-4d74-957a-2e6323f0c241" (UID: "0c05d354-a20f-4d74-957a-2e6323f0c241"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:42:10.918748 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:10.918746    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-api-access-x7l9x" (OuterVolumeSpecName: "kube-api-access-x7l9x") pod "0c05d354-a20f-4d74-957a-2e6323f0c241" (UID: "0c05d354-a20f-4d74-957a-2e6323f0c241"). InnerVolumeSpecName "kube-api-access-x7l9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:42:11.017978 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.017951    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7l9x\" (UniqueName: \"kubernetes.io/projected/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-api-access-x7l9x\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:42:11.017978 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.017974    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:42:11.017978 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.017983    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c05d354-a20f-4d74-957a-2e6323f0c241-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:42:11.018170 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.017992    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c05d354-a20f-4d74-957a-2e6323f0c241-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:42:11.018170 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.018002    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c05d354-a20f-4d74-957a-2e6323f0c241-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:42:11.677503 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.677464    2559 generic.go:356] "Generic (PLEG): container finished" podID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerID="3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc" exitCode=139
Sep 18 19:42:11.677665 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.677517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" event={"ID":"0c05d354-a20f-4d74-957a-2e6323f0c241","Type":"ContainerDied","Data":"3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc"}
Sep 18 19:42:11.677665 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.677536    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"
Sep 18 19:42:11.677665 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.677609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl" event={"ID":"0c05d354-a20f-4d74-957a-2e6323f0c241","Type":"ContainerDied","Data":"7dbaffeb10e85e231b00e4988551a9c1a52d11171fe3689e1e1d7451331ef930"}
Sep 18 19:42:11.677665 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.677650    2559 scope.go:117] "RemoveContainer" containerID="ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7"
Sep 18 19:42:11.686487 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.686460    2559 scope.go:117] "RemoveContainer" containerID="3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc"
Sep 18 19:42:11.695707 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.695683    2559 scope.go:117] "RemoveContainer" containerID="6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c"
Sep 18 19:42:11.702217 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.702179    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"]
Sep 18 19:42:11.703649 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.703635    2559 scope.go:117] "RemoveContainer" containerID="ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7"
Sep 18 19:42:11.703968 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:42:11.703939    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7\": container with ID starting with ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7 not found: ID does not exist" containerID="ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7"
Sep 18 19:42:11.704073 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.703980    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7"} err="failed to get container status \"ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7\": rpc error: code = NotFound desc = could not find container \"ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7\": container with ID starting with ef60a4d5b365d69289a51e03d5923fe86c68d549e363918ec781601a771712b7 not found: ID does not exist"
Sep 18 19:42:11.704073 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.704010    2559 scope.go:117] "RemoveContainer" containerID="3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc"
Sep 18 19:42:11.704371 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:42:11.704350    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc\": container with ID starting with 3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc not found: ID does not exist" containerID="3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc"
Sep 18 19:42:11.704468 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.704379    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc"} err="failed to get container status \"3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc\": rpc error: code = NotFound desc = could not find container \"3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc\": container with ID starting with 3045e58b84d810231b1d613ae8b015ac413273a1a83efab54c92a523e4bfafbc not found: ID does not exist"
Sep 18 19:42:11.704468 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.704400    2559 scope.go:117] "RemoveContainer" containerID="6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c"
Sep 18 19:42:11.704724 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:42:11.704699    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c\": container with ID starting with 6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c not found: ID does not exist" containerID="6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c"
Sep 18 19:42:11.704810 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.704732    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c"} err="failed to get container status \"6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c\": rpc error: code = NotFound desc = could not find container \"6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c\": container with ID starting with 6d0cc92ddcfa791c1a2558c92a8201b664be2d67028f782d8ec82af9eec7355c not found: ID does not exist"
Sep 18 19:42:11.709952 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:11.709934    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-xczvl"]
Sep 18 19:42:13.709748 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:13.709719    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" path="/var/lib/kubelet/pods/0c05d354-a20f-4d74-957a-2e6323f0c241/volumes"
Sep 18 19:42:14.182420 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.182381    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"]
Sep 18 19:42:14.182741 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.182729    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="storage-initializer"
Sep 18 19:42:14.182790 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.182743    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="storage-initializer"
Sep 18 19:42:14.182790 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.182763    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kube-rbac-proxy"
Sep 18 19:42:14.182790 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.182769    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kube-rbac-proxy"
Sep 18 19:42:14.182790 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.182777    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container"
Sep 18 19:42:14.182790 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.182783    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container"
Sep 18 19:42:14.182953 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.182854    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kserve-container"
Sep 18 19:42:14.182953 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.182867    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c05d354-a20f-4d74-957a-2e6323f0c241" containerName="kube-rbac-proxy"
Sep 18 19:42:14.186031 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.186013    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.188918 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.188898    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ct9wb\""
Sep 18 19:42:14.189957 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.189937    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:42:14.190047 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.190001    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:42:14.190047 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.190034    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 19:42:14.190047 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.190039    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:42:14.190270 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.190080    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 19:42:14.190270 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.190139    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:42:14.199169 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.199149    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"]
Sep 18 19:42:14.244964 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.244932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncb4c\" (UniqueName: \"kubernetes.io/projected/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-api-access-ncb4c\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.245111 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.244983    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.245111 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.245031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb102668-4a68-4e29-9337-b3f1a2c861f9-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.245111 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.245055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.245111 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.245078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb102668-4a68-4e29-9337-b3f1a2c861f9-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.346265 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.346233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.346412 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.346288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb102668-4a68-4e29-9337-b3f1a2c861f9-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.346412 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.346326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.346412 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.346359    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb102668-4a68-4e29-9337-b3f1a2c861f9-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.346562 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.346410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncb4c\" (UniqueName: \"kubernetes.io/projected/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-api-access-ncb4c\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.346777 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.346755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb102668-4a68-4e29-9337-b3f1a2c861f9-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.347023 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.346993    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.347126 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.347046    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.348788 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.348774    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb102668-4a68-4e29-9337-b3f1a2c861f9-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.365695 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.365669    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncb4c\" (UniqueName: \"kubernetes.io/projected/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-api-access-ncb4c\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.496474 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.496381    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:14.768288 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:14.768266    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"]
Sep 18 19:42:14.770892 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:42:14.770870    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb102668_4a68_4e29_9337_b3f1a2c861f9.slice/crio-06357a07525be5bd8a022f8e3d578965d67797c18ca14a3b37d2b008f59a400d WatchSource:0}: Error finding container 06357a07525be5bd8a022f8e3d578965d67797c18ca14a3b37d2b008f59a400d: Status 404 returned error can't find the container with id 06357a07525be5bd8a022f8e3d578965d67797c18ca14a3b37d2b008f59a400d
Sep 18 19:42:15.697327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:15.697291    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" event={"ID":"fb102668-4a68-4e29-9337-b3f1a2c861f9","Type":"ContainerStarted","Data":"6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60"}
Sep 18 19:42:15.697469 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:15.697351    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" event={"ID":"fb102668-4a68-4e29-9337-b3f1a2c861f9","Type":"ContainerStarted","Data":"06357a07525be5bd8a022f8e3d578965d67797c18ca14a3b37d2b008f59a400d"}
Sep 18 19:42:19.716156 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:19.716125    2559 generic.go:356] "Generic (PLEG): container finished" podID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerID="6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60" exitCode=0
Sep 18 19:42:19.716600 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:19.716219    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" event={"ID":"fb102668-4a68-4e29-9337-b3f1a2c861f9","Type":"ContainerDied","Data":"6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60"}
Sep 18 19:42:20.722240 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:20.722186    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" event={"ID":"fb102668-4a68-4e29-9337-b3f1a2c861f9","Type":"ContainerStarted","Data":"81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142"}
Sep 18 19:42:20.722635 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:20.722261    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" event={"ID":"fb102668-4a68-4e29-9337-b3f1a2c861f9","Type":"ContainerStarted","Data":"1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475"}
Sep 18 19:42:20.722635 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:20.722560    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:20.722635 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:20.722601    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:20.724050 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:20.724027    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:42:20.759294 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:20.759247    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podStartSLOduration=2.768503613 podStartE2EDuration="6.759233025s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.990729412s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:20.757790465 +0000 UTC m=+1663.211928642" watchObservedRunningTime="2026-09-18 19:42:20.759233025 +0000 UTC m=+1663.213371201"
Sep 18 19:42:21.726254 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:21.726222    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:42:26.731178 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:26.731150    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:42:26.731640 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:26.731615    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:42:36.732485 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:36.732438    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:42:46.732270 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:46.732227    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:42:56.731688 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:42:56.731612    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:43:06.732404 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:06.732362    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:43:16.732245 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:16.732190    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:43:26.732357 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:26.732320    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:43:36.732971 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:36.732936    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:43:44.846565 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:44.846535    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"]
Sep 18 19:43:44.846938 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:44.846742    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" containerID="cri-o://1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475" gracePeriod=30
Sep 18 19:43:44.846938 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:44.846770    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kube-rbac-proxy" containerID="cri-o://81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142" gracePeriod=30
Sep 18 19:43:45.075788 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:45.075752    2559 generic.go:356] "Generic (PLEG): container finished" podID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerID="81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142" exitCode=2
Sep 18 19:43:45.075946 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:45.075826    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" event={"ID":"fb102668-4a68-4e29-9337-b3f1a2c861f9","Type":"ContainerDied","Data":"81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142"}
Sep 18 19:43:46.726436 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:46.726399    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 18 19:43:46.731748 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:46.731718    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:43:49.588420 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.588397    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:43:49.660905 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.660860    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-rbac-proxy-sar-config\") pod \"fb102668-4a68-4e29-9337-b3f1a2c861f9\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") "
Sep 18 19:43:49.660905 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.660912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb102668-4a68-4e29-9337-b3f1a2c861f9-kserve-provision-location\") pod \"fb102668-4a68-4e29-9337-b3f1a2c861f9\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") "
Sep 18 19:43:49.661150 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.661084    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb102668-4a68-4e29-9337-b3f1a2c861f9-proxy-tls\") pod \"fb102668-4a68-4e29-9337-b3f1a2c861f9\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") "
Sep 18 19:43:49.661150 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.661125    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncb4c\" (UniqueName: \"kubernetes.io/projected/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-api-access-ncb4c\") pod \"fb102668-4a68-4e29-9337-b3f1a2c861f9\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") "
Sep 18 19:43:49.661150 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.661147    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-cabundle-cert\") pod \"fb102668-4a68-4e29-9337-b3f1a2c861f9\" (UID: \"fb102668-4a68-4e29-9337-b3f1a2c861f9\") "
Sep 18 19:43:49.661330 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.661244    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb102668-4a68-4e29-9337-b3f1a2c861f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb102668-4a68-4e29-9337-b3f1a2c861f9" (UID: "fb102668-4a68-4e29-9337-b3f1a2c861f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:49.661330 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.661274    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb102668-4a68-4e29-9337-b3f1a2c861f9" (UID: "fb102668-4a68-4e29-9337-b3f1a2c861f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:49.661541 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.661519    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb102668-4a68-4e29-9337-b3f1a2c861f9" (UID: "fb102668-4a68-4e29-9337-b3f1a2c861f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:49.661607 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.661523    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:43:49.661607 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.661564    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb102668-4a68-4e29-9337-b3f1a2c861f9-kserve-provision-location\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:43:49.663118 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.663096    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb102668-4a68-4e29-9337-b3f1a2c861f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb102668-4a68-4e29-9337-b3f1a2c861f9" (UID: "fb102668-4a68-4e29-9337-b3f1a2c861f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:49.663258 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.663232    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-api-access-ncb4c" (OuterVolumeSpecName: "kube-api-access-ncb4c") pod "fb102668-4a68-4e29-9337-b3f1a2c861f9" (UID: "fb102668-4a68-4e29-9337-b3f1a2c861f9"). InnerVolumeSpecName "kube-api-access-ncb4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:43:49.762952 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.762928    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb102668-4a68-4e29-9337-b3f1a2c861f9-proxy-tls\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:43:49.762952 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.762952    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncb4c\" (UniqueName: \"kubernetes.io/projected/fb102668-4a68-4e29-9337-b3f1a2c861f9-kube-api-access-ncb4c\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:43:49.763118 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:49.762961    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb102668-4a68-4e29-9337-b3f1a2c861f9-cabundle-cert\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:43:50.097788 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.097758    2559 generic.go:356] "Generic (PLEG): container finished" podID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerID="1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475" exitCode=0
Sep 18 19:43:50.097969 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.097817    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"
Sep 18 19:43:50.097969 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.097871    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" event={"ID":"fb102668-4a68-4e29-9337-b3f1a2c861f9","Type":"ContainerDied","Data":"1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475"}
Sep 18 19:43:50.097969 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.097924    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n" event={"ID":"fb102668-4a68-4e29-9337-b3f1a2c861f9","Type":"ContainerDied","Data":"06357a07525be5bd8a022f8e3d578965d67797c18ca14a3b37d2b008f59a400d"}
Sep 18 19:43:50.097969 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.097953    2559 scope.go:117] "RemoveContainer" containerID="81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142"
Sep 18 19:43:50.107617 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.107587    2559 scope.go:117] "RemoveContainer" containerID="1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475"
Sep 18 19:43:50.115025 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.115005    2559 scope.go:117] "RemoveContainer" containerID="6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60"
Sep 18 19:43:50.122196 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.122177    2559 scope.go:117] "RemoveContainer" containerID="81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142"
Sep 18 19:43:50.122483 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:43:50.122447    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142\": container with ID starting with 81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142 not found: ID does not exist" containerID="81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142"
Sep 18 19:43:50.122541 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.122492    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142"} err="failed to get container status \"81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142\": rpc error: code = NotFound desc = could not find container \"81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142\": container with ID starting with 81043fc5bded456fdb6c4c0bf9f65f3a4cddcd33ed9abb1a9b98ee79cd972142 not found: ID does not exist"
Sep 18 19:43:50.122541 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.122508    2559 scope.go:117] "RemoveContainer" containerID="1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475"
Sep 18 19:43:50.122768 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:43:50.122745    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475\": container with ID starting with 1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475 not found: ID does not exist" containerID="1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475"
Sep 18 19:43:50.122866 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.122772    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475"} err="failed to get container status \"1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475\": rpc error: code = NotFound desc = could not find container \"1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475\": container with ID starting with 1bd678b01875f6b426e0b9591f409ff0824802fbc68ad140ae5541ea4d658475 not found: ID does not exist"
Sep 18 19:43:50.122866 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.122790    2559 scope.go:117] "RemoveContainer" containerID="6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60"
Sep 18 19:43:50.123036 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:43:50.123021    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60\": container with ID starting with 6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60 not found: ID does not exist" containerID="6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60"
Sep 18 19:43:50.123077 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.123038    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60"} err="failed to get container status \"6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60\": rpc error: code = NotFound desc = could not find container \"6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60\": container with ID starting with 6ef8dd328d5b876e3b6cfd272fa80247a562f0d97ea4e28eb96daa615e8cbe60 not found: ID does not exist"
Sep 18 19:43:50.126645 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.126624    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"]
Sep 18 19:43:50.134030 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:50.134007    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-fsm2n"]
Sep 18 19:43:51.709970 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:43:51.709936    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" path="/var/lib/kubelet/pods/fb102668-4a68-4e29-9337-b3f1a2c861f9/volumes"
Sep 18 19:44:39.746675 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:44:39.746648    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:44:39.751740 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:44:39.751713    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:45:35.738736 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.738702    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kgkjb/must-gather-gvg7r"]
Sep 18 19:45:35.739238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.739022    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="storage-initializer"
Sep 18 19:45:35.739238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.739034    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="storage-initializer"
Sep 18 19:45:35.739238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.739049    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container"
Sep 18 19:45:35.739238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.739055    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container"
Sep 18 19:45:35.739238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.739066    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kube-rbac-proxy"
Sep 18 19:45:35.739238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.739072    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kube-rbac-proxy"
Sep 18 19:45:35.739238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.739128    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kube-rbac-proxy"
Sep 18 19:45:35.739238 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.739134    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb102668-4a68-4e29-9337-b3f1a2c861f9" containerName="kserve-container"
Sep 18 19:45:35.742274 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.742258    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:45:35.745074 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.745040    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kgkjb\"/\"kube-root-ca.crt\""
Sep 18 19:45:35.745401 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.745387    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kgkjb\"/\"openshift-service-ca.crt\""
Sep 18 19:45:35.745770 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.745753    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kgkjb\"/\"default-dockercfg-44hdj\""
Sep 18 19:45:35.751846 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.751824    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kgkjb/must-gather-gvg7r"]
Sep 18 19:45:35.754005 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.753985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fcbc5ce-d878-4d44-a158-97640706b443-must-gather-output\") pod \"must-gather-gvg7r\" (UID: \"0fcbc5ce-d878-4d44-a158-97640706b443\") " pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:45:35.754084 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.754035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrlr\" (UniqueName: \"kubernetes.io/projected/0fcbc5ce-d878-4d44-a158-97640706b443-kube-api-access-9rrlr\") pod \"must-gather-gvg7r\" (UID: \"0fcbc5ce-d878-4d44-a158-97640706b443\") " pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:45:35.855180 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.855149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fcbc5ce-d878-4d44-a158-97640706b443-must-gather-output\") pod \"must-gather-gvg7r\" (UID: \"0fcbc5ce-d878-4d44-a158-97640706b443\") " pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:45:35.855400 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.855188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrlr\" (UniqueName: \"kubernetes.io/projected/0fcbc5ce-d878-4d44-a158-97640706b443-kube-api-access-9rrlr\") pod \"must-gather-gvg7r\" (UID: \"0fcbc5ce-d878-4d44-a158-97640706b443\") " pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:45:35.855543 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.855523    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fcbc5ce-d878-4d44-a158-97640706b443-must-gather-output\") pod \"must-gather-gvg7r\" (UID: \"0fcbc5ce-d878-4d44-a158-97640706b443\") " pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:45:35.864730 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:35.864705    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrlr\" (UniqueName: \"kubernetes.io/projected/0fcbc5ce-d878-4d44-a158-97640706b443-kube-api-access-9rrlr\") pod \"must-gather-gvg7r\" (UID: \"0fcbc5ce-d878-4d44-a158-97640706b443\") " pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:45:36.062461 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:36.062425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:45:36.332255 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:36.332229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kgkjb/must-gather-gvg7r"]
Sep 18 19:45:36.334400 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:45:36.334374    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fcbc5ce_d878_4d44_a158_97640706b443.slice/crio-542360374e76f8f3a39a50eb1860081309ad9ba3247e25fbb042322dfbf9469c WatchSource:0}: Error finding container 542360374e76f8f3a39a50eb1860081309ad9ba3247e25fbb042322dfbf9469c: Status 404 returned error can't find the container with id 542360374e76f8f3a39a50eb1860081309ad9ba3247e25fbb042322dfbf9469c
Sep 18 19:45:36.336645 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:36.336630    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:45:36.545911 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:36.545881    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgkjb/must-gather-gvg7r" event={"ID":"0fcbc5ce-d878-4d44-a158-97640706b443","Type":"ContainerStarted","Data":"542360374e76f8f3a39a50eb1860081309ad9ba3247e25fbb042322dfbf9469c"}
Sep 18 19:45:43.582358 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:43.582318    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgkjb/must-gather-gvg7r" event={"ID":"0fcbc5ce-d878-4d44-a158-97640706b443","Type":"ContainerStarted","Data":"8b0441dde8a4f3ca553e9e5b80f7c726baef579028e91fa7d07b07ab9854da99"}
Sep 18 19:45:43.582715 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:43.582367    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgkjb/must-gather-gvg7r" event={"ID":"0fcbc5ce-d878-4d44-a158-97640706b443","Type":"ContainerStarted","Data":"2de3ebb3c37266232ad676c8ae881d8cd7f3fd865e799ff6d935f63694964b17"}
Sep 18 19:45:43.601987 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:45:43.601936    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kgkjb/must-gather-gvg7r" podStartSLOduration=2.275938609 podStartE2EDuration="8.601919122s" totalImagesPullingTime="6.325980513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:43.601284068 +0000 UTC m=+1866.055422251" watchObservedRunningTime="2026-09-18 19:45:43.601919122 +0000 UTC m=+1866.056057327"
Sep 18 19:46:06.693807 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:06.693776    2559 generic.go:356] "Generic (PLEG): container finished" podID="0fcbc5ce-d878-4d44-a158-97640706b443" containerID="2de3ebb3c37266232ad676c8ae881d8cd7f3fd865e799ff6d935f63694964b17" exitCode=0
Sep 18 19:46:06.694232 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:06.693850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgkjb/must-gather-gvg7r" event={"ID":"0fcbc5ce-d878-4d44-a158-97640706b443","Type":"ContainerDied","Data":"2de3ebb3c37266232ad676c8ae881d8cd7f3fd865e799ff6d935f63694964b17"}
Sep 18 19:46:06.694232 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:06.694217    2559 scope.go:117] "RemoveContainer" containerID="2de3ebb3c37266232ad676c8ae881d8cd7f3fd865e799ff6d935f63694964b17"
Sep 18 19:46:06.940083 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:06.940052    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kgkjb_must-gather-gvg7r_0fcbc5ce-d878-4d44-a158-97640706b443/gather/0.log"
Sep 18 19:46:10.613794 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:10.613762    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pxfxm_edf078c1-1bad-4f23-8753-bd4414735c17/global-pull-secret-syncer/0.log"
Sep 18 19:46:10.703109 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:10.703080    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6lhsw_273ddf75-7335-47f7-9bc7-c0c3f4727ebd/konnectivity-agent/0.log"
Sep 18 19:46:10.786742 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:10.786713    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-38.ec2.internal_057a0ec2fb9c26d68b86efbdbfc5dfe9/haproxy/0.log"
Sep 18 19:46:12.528010 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.527973    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-kgkjb/must-gather-gvg7r"]
Sep 18 19:46:12.528549 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.528224    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-kgkjb/must-gather-gvg7r" podUID="0fcbc5ce-d878-4d44-a158-97640706b443" containerName="copy" containerID="cri-o://8b0441dde8a4f3ca553e9e5b80f7c726baef579028e91fa7d07b07ab9854da99" gracePeriod=2
Sep 18 19:46:12.530369 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.530346    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-kgkjb/must-gather-gvg7r"]
Sep 18 19:46:12.530727 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:46:12.530704    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gvg7r\" is forbidden: User \"system:node:ip-10-0-138-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kgkjb\": no relationship found between node 'ip-10-0-138-38.ec2.internal' and this object" podUID="0fcbc5ce-d878-4d44-a158-97640706b443" pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:46:12.719636 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.719614    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kgkjb_must-gather-gvg7r_0fcbc5ce-d878-4d44-a158-97640706b443/copy/0.log"
Sep 18 19:46:12.720010 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.719983    2559 generic.go:356] "Generic (PLEG): container finished" podID="0fcbc5ce-d878-4d44-a158-97640706b443" containerID="8b0441dde8a4f3ca553e9e5b80f7c726baef579028e91fa7d07b07ab9854da99" exitCode=143
Sep 18 19:46:12.756253 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.756235    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kgkjb_must-gather-gvg7r_0fcbc5ce-d878-4d44-a158-97640706b443/copy/0.log"
Sep 18 19:46:12.757989 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.757972    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:46:12.761564 ip-10-0-138-38 kubenswrapper[2559]: E0918 19:46:12.761543    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gvg7r\" is forbidden: User \"system:node:ip-10-0-138-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kgkjb\": no relationship found between node 'ip-10-0-138-38.ec2.internal' and this object" podUID="0fcbc5ce-d878-4d44-a158-97640706b443" pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:46:12.793020 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.792972    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fcbc5ce-d878-4d44-a158-97640706b443-must-gather-output\") pod \"0fcbc5ce-d878-4d44-a158-97640706b443\" (UID: \"0fcbc5ce-d878-4d44-a158-97640706b443\") "
Sep 18 19:46:12.793020 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.793012    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rrlr\" (UniqueName: \"kubernetes.io/projected/0fcbc5ce-d878-4d44-a158-97640706b443-kube-api-access-9rrlr\") pod \"0fcbc5ce-d878-4d44-a158-97640706b443\" (UID: \"0fcbc5ce-d878-4d44-a158-97640706b443\") "
Sep 18 19:46:12.795055 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.795029    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcbc5ce-d878-4d44-a158-97640706b443-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0fcbc5ce-d878-4d44-a158-97640706b443" (UID: "0fcbc5ce-d878-4d44-a158-97640706b443"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:46:12.795155 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.795133    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcbc5ce-d878-4d44-a158-97640706b443-kube-api-access-9rrlr" (OuterVolumeSpecName: "kube-api-access-9rrlr") pod "0fcbc5ce-d878-4d44-a158-97640706b443" (UID: "0fcbc5ce-d878-4d44-a158-97640706b443"). InnerVolumeSpecName "kube-api-access-9rrlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:46:12.893646 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.893612    2559 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fcbc5ce-d878-4d44-a158-97640706b443-must-gather-output\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:46:12.893646 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:12.893641    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrlr\" (UniqueName: \"kubernetes.io/projected/0fcbc5ce-d878-4d44-a158-97640706b443-kube-api-access-9rrlr\") on node \"ip-10-0-138-38.ec2.internal\" DevicePath \"\""
Sep 18 19:46:13.710143 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:13.710111    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fcbc5ce-d878-4d44-a158-97640706b443" path="/var/lib/kubelet/pods/0fcbc5ce-d878-4d44-a158-97640706b443/volumes"
Sep 18 19:46:13.725172 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:13.725152    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kgkjb_must-gather-gvg7r_0fcbc5ce-d878-4d44-a158-97640706b443/copy/0.log"
Sep 18 19:46:13.725516 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:13.725501    2559 scope.go:117] "RemoveContainer" containerID="8b0441dde8a4f3ca553e9e5b80f7c726baef579028e91fa7d07b07ab9854da99"
Sep 18 19:46:13.725575 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:13.725534    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgkjb/must-gather-gvg7r"
Sep 18 19:46:13.733090 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:13.733073    2559 scope.go:117] "RemoveContainer" containerID="2de3ebb3c37266232ad676c8ae881d8cd7f3fd865e799ff6d935f63694964b17"
Sep 18 19:46:14.812164 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:14.812137    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ca01eda-ec1d-4f62-a1ff-e59798842c65/alertmanager/0.log"
Sep 18 19:46:14.847746 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:14.847722    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ca01eda-ec1d-4f62-a1ff-e59798842c65/config-reloader/0.log"
Sep 18 19:46:14.879111 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:14.879036    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ca01eda-ec1d-4f62-a1ff-e59798842c65/kube-rbac-proxy-web/0.log"
Sep 18 19:46:14.917935 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:14.917911    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ca01eda-ec1d-4f62-a1ff-e59798842c65/kube-rbac-proxy/0.log"
Sep 18 19:46:14.957249 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:14.957227    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ca01eda-ec1d-4f62-a1ff-e59798842c65/kube-rbac-proxy-metric/0.log"
Sep 18 19:46:14.992406 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:14.992385    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ca01eda-ec1d-4f62-a1ff-e59798842c65/prom-label-proxy/0.log"
Sep 18 19:46:15.023748 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.023730    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ca01eda-ec1d-4f62-a1ff-e59798842c65/init-config-reloader/0.log"
Sep 18 19:46:15.313600 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.313576    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2d9kz_e476d1f0-b191-403c-9e39-db73496b2834/node-exporter/0.log"
Sep 18 19:46:15.348327 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.348297    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2d9kz_e476d1f0-b191-403c-9e39-db73496b2834/kube-rbac-proxy/0.log"
Sep 18 19:46:15.369652 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.369630    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2d9kz_e476d1f0-b191-403c-9e39-db73496b2834/init-textfile/0.log"
Sep 18 19:46:15.744490 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.744464    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8bbbeb7c-05f6-45f1-9a03-9e74ac903bab/prometheus/0.log"
Sep 18 19:46:15.771095 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.771076    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8bbbeb7c-05f6-45f1-9a03-9e74ac903bab/config-reloader/0.log"
Sep 18 19:46:15.805278 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.805255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8bbbeb7c-05f6-45f1-9a03-9e74ac903bab/thanos-sidecar/0.log"
Sep 18 19:46:15.840005 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.839983    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8bbbeb7c-05f6-45f1-9a03-9e74ac903bab/kube-rbac-proxy-web/0.log"
Sep 18 19:46:15.880362 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.880344    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8bbbeb7c-05f6-45f1-9a03-9e74ac903bab/kube-rbac-proxy/0.log"
Sep 18 19:46:15.913235 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.913190    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8bbbeb7c-05f6-45f1-9a03-9e74ac903bab/kube-rbac-proxy-thanos/0.log"
Sep 18 19:46:15.944153 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:15.944086    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8bbbeb7c-05f6-45f1-9a03-9e74ac903bab/init-config-reloader/0.log"
Sep 18 19:46:16.061776 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:16.061749    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-7lghg_caf43189-4ad3-42c5-81c2-580ec4ed8956/prometheus-operator-admission-webhook/0.log"
Sep 18 19:46:16.114747 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:16.114724    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fcbf478c9-gtx5k_a6cb263b-7557-4f6f-8601-f31c0b16fe54/telemeter-client/0.log"
Sep 18 19:46:16.174644 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:16.174615    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fcbf478c9-gtx5k_a6cb263b-7557-4f6f-8601-f31c0b16fe54/reload/0.log"
Sep 18 19:46:16.254904 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:16.254879    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fcbf478c9-gtx5k_a6cb263b-7557-4f6f-8601-f31c0b16fe54/kube-rbac-proxy/0.log"
Sep 18 19:46:16.345066 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:16.345044    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74478c4dc8-9s9s5_ff591b00-0c33-469a-9d64-c9aca37b5830/thanos-query/0.log"
Sep 18 19:46:16.403324 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:16.403298    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74478c4dc8-9s9s5_ff591b00-0c33-469a-9d64-c9aca37b5830/kube-rbac-proxy-web/0.log"
Sep 18 19:46:16.457777 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:16.457714    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74478c4dc8-9s9s5_ff591b00-0c33-469a-9d64-c9aca37b5830/kube-rbac-proxy/0.log"
Sep 18 19:46:16.494377 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:16.494349    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74478c4dc8-9s9s5_ff591b00-0c33-469a-9d64-c9aca37b5830/prom-label-proxy/0.log"
Sep 18 19:46:16.540667 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:16.540648    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74478c4dc8-9s9s5_ff591b00-0c33-469a-9d64-c9aca37b5830/kube-rbac-proxy-rules/0.log"
Sep 18 19:46:16.585112 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:16.585089    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74478c4dc8-9s9s5_ff591b00-0c33-469a-9d64-c9aca37b5830/kube-rbac-proxy-metrics/0.log"
Sep 18 19:46:17.775119 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:17.775094    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-bqhhx_048b4230-fb95-4853-8eaf-830198c66298/networking-console-plugin/0.log"
Sep 18 19:46:18.206776 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.206743    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"]
Sep 18 19:46:18.207092 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.207078    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fcbc5ce-d878-4d44-a158-97640706b443" containerName="gather"
Sep 18 19:46:18.207137 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.207094    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fcbc5ce-d878-4d44-a158-97640706b443" containerName="gather"
Sep 18 19:46:18.207137 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.207109    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fcbc5ce-d878-4d44-a158-97640706b443" containerName="copy"
Sep 18 19:46:18.207137 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.207114    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fcbc5ce-d878-4d44-a158-97640706b443" containerName="copy"
Sep 18 19:46:18.207246 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.207175    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fcbc5ce-d878-4d44-a158-97640706b443" containerName="copy"
Sep 18 19:46:18.207246 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.207185    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fcbc5ce-d878-4d44-a158-97640706b443" containerName="gather"
Sep 18 19:46:18.211060 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.211044    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.213444 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.213425    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hzj7l\"/\"default-dockercfg-4b7ng\""
Sep 18 19:46:18.213770 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.213757    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hzj7l\"/\"openshift-service-ca.crt\""
Sep 18 19:46:18.214682 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.214664    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hzj7l\"/\"kube-root-ca.crt\""
Sep 18 19:46:18.218588 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.218567    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"]
Sep 18 19:46:18.229193 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.229171    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-proc\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.229307 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.229266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-sys\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.229350 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.229308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-lib-modules\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.229350 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.229344    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-podres\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.229452 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.229434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n5qd\" (UniqueName: \"kubernetes.io/projected/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-kube-api-access-9n5qd\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.280424 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.280402    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/1.log"
Sep 18 19:46:18.285097 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.285076    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-b9dcv_cc94edce-8cdd-40dc-bd9e-68cc15000b6a/console-operator/2.log"
Sep 18 19:46:18.330402 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.330376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-sys\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.330494 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.330405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-lib-modules\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.330494 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.330426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-podres\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.330578 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.330499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-sys\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.330578 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.330503    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n5qd\" (UniqueName: \"kubernetes.io/projected/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-kube-api-access-9n5qd\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.330578 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.330552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-proc\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.330578 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.330560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-lib-modules\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.330714 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.330610    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-podres\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.330714 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.330628    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-proc\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.339954 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.339933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n5qd\" (UniqueName: \"kubernetes.io/projected/32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2-kube-api-access-9n5qd\") pod \"perf-node-gather-daemonset-rpq6l\" (UID: \"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.521391 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.521361    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:18.792454 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:18.792431    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"]
Sep 18 19:46:18.794291 ip-10-0-138-38 kubenswrapper[2559]: W0918 19:46:18.794262    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod32392a4e_7da3_43c7_8f4c_9ae1b1dd5be2.slice/crio-5d91f60a064474e9e15f0477c5423cf94ba2f8a1d971a3097f32d24b460aa622 WatchSource:0}: Error finding container 5d91f60a064474e9e15f0477c5423cf94ba2f8a1d971a3097f32d24b460aa622: Status 404 returned error can't find the container with id 5d91f60a064474e9e15f0477c5423cf94ba2f8a1d971a3097f32d24b460aa622
Sep 18 19:46:19.750678 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:19.750637    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l" event={"ID":"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2","Type":"ContainerStarted","Data":"3684d9913551ed61cb6fcb87e7bccf4ec03c6a968192b14d639b24382e986506"}
Sep 18 19:46:19.750866 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:19.750692    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:19.750866 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:19.750715    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l" event={"ID":"32392a4e-7da3-43c7-8f4c-9ae1b1dd5be2","Type":"ContainerStarted","Data":"5d91f60a064474e9e15f0477c5423cf94ba2f8a1d971a3097f32d24b460aa622"}
Sep 18 19:46:19.771571 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:19.771526    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l" podStartSLOduration=1.771511944 podStartE2EDuration="1.771511944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:19.769599558 +0000 UTC m=+1902.223737759" watchObservedRunningTime="2026-09-18 19:46:19.771511944 +0000 UTC m=+1902.225650162"
Sep 18 19:46:20.376214 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:20.376182    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fqqh8_fbe654a7-76ea-4741-ae34-d2b4566cf622/dns/0.log"
Sep 18 19:46:20.419033 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:20.419011    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fqqh8_fbe654a7-76ea-4741-ae34-d2b4566cf622/kube-rbac-proxy/0.log"
Sep 18 19:46:20.460908 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:20.460885    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-67hqf_af4fccc9-2cf5-42f5-aa77-d7bc375a4c49/dns-node-resolver/0.log"
Sep 18 19:46:21.147350 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:21.147317    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xfkrz_76d77ab0-46a5-432c-a812-67a2e57e0f07/node-ca/0.log"
Sep 18 19:46:21.890089 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:21.890057    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58ccb95cb-jb24t_51a4f6b3-247d-4614-976d-e585479998d7/router/0.log"
Sep 18 19:46:22.256563 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:22.256483    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vq6nl_4af3fc5f-2070-42ee-a588-c4197598b28a/serve-healthcheck-canary/0.log"
Sep 18 19:46:22.634703 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:22.634673    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-kfskl_b6899294-bd1e-4b9c-ab8a-f10dfaba7a94/insights-operator/1.log"
Sep 18 19:46:22.634880 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:22.634780    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-kfskl_b6899294-bd1e-4b9c-ab8a-f10dfaba7a94/insights-operator/0.log"
Sep 18 19:46:22.729600 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:22.729573    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7pvzt_805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2/kube-rbac-proxy/0.log"
Sep 18 19:46:22.755344 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:22.755320    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7pvzt_805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2/exporter/0.log"
Sep 18 19:46:22.786169 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:22.786144    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7pvzt_805bdd1c-3ae6-4b11-ac88-ed2fc7af00a2/extractor/0.log"
Sep 18 19:46:25.764886 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:25.764860    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-rpq6l"
Sep 18 19:46:30.035024 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:30.034995    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-sktcf_59dee8c0-41d0-414e-b17b-99f7522088f9/migrator/0.log"
Sep 18 19:46:30.060135 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:30.060111    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-sktcf_59dee8c0-41d0-414e-b17b-99f7522088f9/graceful-termination/0.log"
Sep 18 19:46:30.464174 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:30.464093    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-92gfp_fbe7c860-1b27-480f-9de5-ba6eddfb76a0/kube-storage-version-migrator-operator/1.log"
Sep 18 19:46:30.465033 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:30.465016    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-92gfp_fbe7c860-1b27-480f-9de5-ba6eddfb76a0/kube-storage-version-migrator-operator/0.log"
Sep 18 19:46:31.794436 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:31.794405    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-87xpd_42c37c95-3249-4102-a12a-8d903597aee0/kube-multus/0.log"
Sep 18 19:46:32.030904 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:32.030875    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lhfb5_d5cb6368-a50c-4ec2-b30e-30effe2cf405/kube-multus-additional-cni-plugins/0.log"
Sep 18 19:46:32.062081 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:32.062001    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lhfb5_d5cb6368-a50c-4ec2-b30e-30effe2cf405/egress-router-binary-copy/0.log"
Sep 18 19:46:32.090038 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:32.090014    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lhfb5_d5cb6368-a50c-4ec2-b30e-30effe2cf405/cni-plugins/0.log"
Sep 18 19:46:32.117950 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:32.117932    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lhfb5_d5cb6368-a50c-4ec2-b30e-30effe2cf405/bond-cni-plugin/0.log"
Sep 18 19:46:32.152276 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:32.152242    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lhfb5_d5cb6368-a50c-4ec2-b30e-30effe2cf405/routeoverride-cni/0.log"
Sep 18 19:46:32.182601 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:32.182581    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lhfb5_d5cb6368-a50c-4ec2-b30e-30effe2cf405/whereabouts-cni-bincopy/0.log"
Sep 18 19:46:32.214605 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:32.214587    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lhfb5_d5cb6368-a50c-4ec2-b30e-30effe2cf405/whereabouts-cni/0.log"
Sep 18 19:46:32.625883 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:32.625853    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g9cck_cb350aae-48d3-4913-aec6-cd5f71e51350/network-metrics-daemon/0.log"
Sep 18 19:46:32.654250 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:32.654230    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g9cck_cb350aae-48d3-4913-aec6-cd5f71e51350/kube-rbac-proxy/0.log"
Sep 18 19:46:33.515554 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:33.515527    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hfmt_a1d92004-612a-4adc-8159-2398bc926270/ovn-controller/0.log"
Sep 18 19:46:33.568580 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:33.568551    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hfmt_a1d92004-612a-4adc-8159-2398bc926270/ovn-acl-logging/0.log"
Sep 18 19:46:33.612496 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:33.612477    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hfmt_a1d92004-612a-4adc-8159-2398bc926270/kube-rbac-proxy-node/0.log"
Sep 18 19:46:33.654782 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:33.654759    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hfmt_a1d92004-612a-4adc-8159-2398bc926270/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 19:46:33.691896 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:33.691873    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hfmt_a1d92004-612a-4adc-8159-2398bc926270/northd/0.log"
Sep 18 19:46:33.745145 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:33.745127    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hfmt_a1d92004-612a-4adc-8159-2398bc926270/nbdb/0.log"
Sep 18 19:46:33.787125 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:33.787106    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hfmt_a1d92004-612a-4adc-8159-2398bc926270/sbdb/0.log"
Sep 18 19:46:33.844877 ip-10-0-138-38 kubenswrapper[2559]: I0918 19:46:33.844856    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hfmt_a1d92004-612a-4adc-8159-2398bc926270/ovnkube-controller/0.log"