Jul 21 17:24:04.824972 ip-10-0-136-31 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 21 17:24:04.824983 ip-10-0-136-31 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 21 17:24:04.824990 ip-10-0-136-31 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 21 17:24:04.825213 ip-10-0-136-31 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 21 17:24:14.892241 ip-10-0-136-31 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 21 17:24:14.892283 ip-10-0-136-31 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ed1ffed72765480d84f490d358e8125d --
Jul 21 17:26:14.319973 ip-10-0-136-31 systemd[1]: Starting Kubernetes Kubelet...
Jul 21 17:26:14.735984 ip-10-0-136-31 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:26:14.735984 ip-10-0-136-31 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 21 17:26:14.735984 ip-10-0-136-31 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:26:14.735984 ip-10-0-136-31 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 21 17:26:14.735984 ip-10-0-136-31 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:26:14.737021 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.736937    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 21 17:26:14.741202 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741187    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:14.741202 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741203    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741206    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741210    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741213    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741216    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741220    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741223    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741226    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741228    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741231    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741233    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741236    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741239    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741241    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741244    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741247    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741250    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741253    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741256    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741259    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:14.741261 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741261    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741264    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741267    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741270    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741273    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741277    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741281    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741284    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741287    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741290    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741292    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741295    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741298    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741301    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741304    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741307    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741310    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741313    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741315    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:14.741742 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741318    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741320    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741323    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741325    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741328    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741330    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741333    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741335    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741338    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741340    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741343    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741345    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741348    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741351    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741353    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741356    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741360    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741363    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741366    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741368    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:14.742198 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741371    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741373    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741376    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741380    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741383    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741386    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741389    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741392    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741395    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741398    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741400    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741404    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741407    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741409    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741412    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741415    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741418    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741421    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741424    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:14.742694 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741426    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741428    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741431    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741434    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741436    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741439    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741441    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741444    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741447    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741450    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741452    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741455    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741458    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741462    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741465    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741468    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741470    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741472    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741475    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.741477    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:14.743144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742173    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742179    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742182    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742185    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742187    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742192    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742196    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742200    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742203    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742206    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742209    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742213    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742217    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742220    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742223    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742226    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742228    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742231    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742234    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:14.743615 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742237    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742239    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742242    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742245    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742248    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742250    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742253    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742255    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742258    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742260    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742263    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742266    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742268    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742270    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742273    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742275    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742278    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742280    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742283    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742286    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:14.744106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742295    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742298    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742301    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742304    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742306    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742309    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742312    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742316    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742319    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742322    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742324    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742327    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742330    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742333    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742336    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742338    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742341    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742344    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742346    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742349    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:14.744595 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742351    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742354    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742357    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742360    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742362    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742365    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742367    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742370    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742373    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742375    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742378    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742380    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742383    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742386    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742388    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742391    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742393    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742397    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742400    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742403    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:14.745082 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742406    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742408    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742410    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742413    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742416    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742418    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742421    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742425    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742428    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742430    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742433    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742435    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742438    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742440    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742443    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742445    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742448    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742450    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742453    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742456    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:14.745569 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742832    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742837    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742840    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742843    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742846    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742849    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742852    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742856    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742859    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742862    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742866    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742869    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742871    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742875    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742878    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742881    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742884    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742887    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742889    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:14.746069 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742893    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742895    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742898    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742900    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742903    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742906    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742908    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742911    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742913    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742916    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742918    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742921    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742924    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742926    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742930    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742934    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742937    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742940    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742943    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:14.746533 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742946    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742949    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742952    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742955    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742957    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742960    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742963    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742966    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742969    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742972    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742975    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742978    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742980    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742983    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742985    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742988    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742991    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742994    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742996    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.742999    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:14.747039 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743003    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743006    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743008    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743012    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743014    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743017    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743019    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743022    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743025    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743027    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743030    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743032    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743035    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743038    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743041    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743044    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743048    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743051    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743054    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743058    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:14.747521 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743061    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743063    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743065    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743068    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743071    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743073    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743076    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743078    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743081    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743083    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743086    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743089    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743092    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743095    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743097    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743100    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743102    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743105    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743107    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743110    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:14.748028 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743112    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743180    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743187    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743194    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743200    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743207    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743212    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743219    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743225    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743228    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743231    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743236    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743239    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743242    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743245    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743248    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743251    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743254    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743257    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743261    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743263    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743266    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743270    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743273    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 21 17:26:14.748535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743278    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743280    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743284    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743287    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743303    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743307    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743311    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743314    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743317    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743322    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743325    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743328    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743331    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743334    2579 flags.go:64] FLAG: --enable-server="true"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743337    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743342    2579 flags.go:64] FLAG: --event-burst="100"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743346    2579 flags.go:64] FLAG: --event-qps="50"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743348    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743351    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743356    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743359    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743362    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743365    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743369    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743371    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 21 17:26:14.749127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743374    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743377    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743380    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743383    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743385    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743388    2579 flags.go:64] FLAG: --feature-gates=""
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743394    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743398    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743402    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743405    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743407    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743410    2579 flags.go:64] FLAG: --help="false"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743413    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-136-31.ec2.internal"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743417    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743419    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743423    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743426    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743429    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743432    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743435    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743438    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743441    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743444    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743447    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 21 17:26:14.749794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743450    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743453    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743459    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743462    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743464    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743467    2579 flags.go:64] FLAG: --lock-file=""
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743470    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743473    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743475    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743481    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743484    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743487    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743490    2579 flags.go:64] FLAG: --logging-format="text"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743493    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743496    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743501    2579 flags.go:64] FLAG: --manifest-url=""
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743504    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743508    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743511    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743515    2579 flags.go:64] FLAG: --max-pods="110"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743518    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743521    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743524    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743527    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743530    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 21 17:26:14.750856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743533    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743535    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743542    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743545    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743548    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743551    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743554    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743560    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743562    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743565    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743569    2579 flags.go:64] FLAG: --port="10250"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743573    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743576    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ec49570b0acf2489"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743579    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743582    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743585    2579 flags.go:64] FLAG: --register-node="true"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743587    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743591    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743594    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743597    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743599    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743603    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743608    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743611    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743613    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 21 17:26:14.751944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743616    2579 flags.go:64] FLAG: --runonce="false"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743619    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743622    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743625    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743628    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743631    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743634    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743637    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743640    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743643    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743646    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743660    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743664    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743667    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743670    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743674    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743680    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743684    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743687    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743691    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743694    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743697    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743699    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743702    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743705    2579 flags.go:64] FLAG: --v="2"
Jul 21 17:26:14.753039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743709    2579 flags.go:64] FLAG: --version="false"
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743713    2579 flags.go:64] FLAG: --vmodule=""
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743716    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.743720    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743834    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743838    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743841    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743844    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743847    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743849    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743852    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743855    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743858    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743860    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743863    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743866    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743869    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743871    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743874    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743877    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743879    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:14.753966 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743882    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743885    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743888    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743892    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743895    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743897    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743900    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743902    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743905    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743907    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743910    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743912    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743915    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743917    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743920    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743925    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743927    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743929    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743932    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743935    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:14.754846 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743937    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743941    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743945    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743948    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743951    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743954    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743956    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743960    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743964    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743966    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743969    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743972    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743975    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743977    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743980    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743984    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743987    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743989    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743992    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:14.755423 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743994    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743997    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.743999    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744002    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744005    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744007    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744010    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744012    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744023    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744027    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744029    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744032    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744034    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744037    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744040    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744042    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744045    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744048    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744050    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:14.755937 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744054    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744057    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744060    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744063    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744066    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744068    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744071    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744074    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744076    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744084    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744087    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744090    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744093    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744096    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744099    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744101    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744104    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744106    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744109    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744112    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:14.756404 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744114    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744118    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744121    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.744123    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.744972    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.752174    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.752193    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.752213    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.752222    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752305    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752313    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752318    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752323    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752328    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:14.757144 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752333    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752337    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752341    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752346    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752351    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752355    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752360    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752364    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752368    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752373    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752377    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752382    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752386    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752390    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752394    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752399    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752404    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752408    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752412    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752417    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:14.757530 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752421    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752425    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752430    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752434    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752440    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752445    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752449    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752453    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752458    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752462    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752466    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752471    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752475    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752479    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752483    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752489    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752493    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752497    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752501    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752505    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:14.758182 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752510    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752514    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752519    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752524    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752528    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752533    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752537    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752541    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752546    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752550    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752554    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752558    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752563    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752567    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752571    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752576    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752581    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752586    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752590    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752594    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:14.758885 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752599    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752603    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752607    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752611    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752618    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752624    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752628    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752633    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752638    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752642    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752646    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752666    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752671    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752675    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752682    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752687    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752692    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752696    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752700    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:14.759446 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752704    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752708    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752712    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752716    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752720    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752724    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752728    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752732    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752736    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752741    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752745    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752750    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752754    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752758    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752765    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:14.759930 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.752773    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752951    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752960    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752965    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752969    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752974    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752981    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752987    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752991    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.752996    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753001    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753005    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753010    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753014    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753018    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753022    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753027    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753031    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753035    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:14.760320 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753040    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753043    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753047    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753051    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753055    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753059    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753063    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753067    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753071    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753075    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753082    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753088    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753092    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753096    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753100    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753104    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753108    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753112    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753116    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753121    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:14.760960 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753125    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753129    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753133    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753137    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753141    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753145    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753149    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753153    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753158    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753163    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753167    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753172    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753176    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753180    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753184    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753188    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753192    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753196    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753200    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753204    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:14.761490 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753208    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753212    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753216    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753224    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753229    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753234    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753238    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753243    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753247    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753252    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753256    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753260    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753265    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753269    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753274    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753278    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753283    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753287    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753291    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:14.762298 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753295    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753299    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753304    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753308    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753312    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753316    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753320    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753324    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753328    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753333    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753337    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753341    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753345    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753349    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753353    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753358    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753362    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753367    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753371    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753376    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:14.762916 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753380    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:14.763384 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:14.753384    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:14.763384 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.753391    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:26:14.763384 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.753531    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 21 17:26:14.763384 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.758964    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 21 17:26:14.763384 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.760298    2579 server.go:1019] "Starting client certificate rotation"
Jul 21 17:26:14.763384 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.760408    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 21 17:26:14.763384 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.760502    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 21 17:26:14.763384 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.762765    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 21 17:26:14.763893 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.763876    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 21 17:26:14.771551 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.771534    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 21 17:26:14.773898 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.773885    2579 log.go:25] "Validated CRI v1 image API"
Jul 21 17:26:14.774700 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.774684    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 21 17:26:14.778982 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.778962    2579 fs.go:135] Filesystem UUIDs: map[761f74bd-2eb7-4e8c-a3cc-70e71be0376b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f67e0807-926f-4c77-b8f5-7e02714d0ab3:/dev/nvme0n1p4]
Jul 21 17:26:14.779044 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.778980    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 21 17:26:14.785580 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.785469    2579 manager.go:216] Machine: {Timestamp:2026-07-21 17:26:14.784071099 +0000 UTC m=+0.342870073 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099695 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2efc80342441eba879bc774c13570b SystemUUID:ec2efc80-3424-41eb-a879-bc774c13570b BootID:ed1ffed7-2765-480d-84f4-90d358e8125d Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fe:f2:0c:78:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fe:f2:0c:78:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:ab:e8:7f:15:22 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 21 17:26:14.785580 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.785576    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 21 17:26:14.785697 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.785665    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 21 17:26:14.789212 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.789183    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 21 17:26:14.789363 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.789212    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 21 17:26:14.790021 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.789998    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 21 17:26:14.790583 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.790571    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 21 17:26:14.790620 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.790586    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 21 17:26:14.790620 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.790594    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 21 17:26:14.791727 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.791717    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 21 17:26:14.792964 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.792951    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 21 17:26:14.795928 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.795916    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 21 17:26:14.796070 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.796059    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 21 17:26:14.796122 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.796088    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 21 17:26:14.796122 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.796104    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 21 17:26:14.796122 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.796122    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 21 17:26:14.796248 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.796146    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 21 17:26:14.796248 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.796166    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 21 17:26:14.796580 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.796568    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 21 17:26:14.796612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.796588    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 21 17:26:14.798752 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.798736    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 21 17:26:14.800557 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.800544    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 21 17:26:14.800600 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.800574    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 21 17:26:14.801568 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801559    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 21 17:26:14.801612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801574    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 21 17:26:14.801612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801580    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 21 17:26:14.801612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801586    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 21 17:26:14.801612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801595    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 21 17:26:14.801612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801601    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 21 17:26:14.801612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801609    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 21 17:26:14.801612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801615    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 21 17:26:14.801807 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801626    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 21 17:26:14.801807 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801632    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 21 17:26:14.801807 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801640    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 21 17:26:14.801807 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.801645    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 21 17:26:14.803256 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.803247    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 21 17:26:14.803288 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.803255    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 21 17:26:14.807603 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.807586    2579 server.go:1268] "Started kubelet"
Jul 21 17:26:14.807755 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.807703    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 21 17:26:14.807831 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.807780    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 21 17:26:14.807885 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.807747    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 21 17:26:14.808536 ip-10-0-136-31 systemd[1]: Started Kubernetes Kubelet.
Jul 21 17:26:14.809274 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.809253    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 21 17:26:14.810492 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.810479    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 21 17:26:14.812094 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.812071    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 21 17:26:14.812189 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.812156    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 21 17:26:14.812275 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.812256    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 21 17:26:14.813212 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.812218    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-31.ec2.internal.18c45e0feb1b8988  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-31.ec2.internal,UID:ip-10-0-136-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-31.ec2.internal,},FirstTimestamp:2026-07-21 17:26:14.807529864 +0000 UTC m=+0.366328839,LastTimestamp:2026-07-21 17:26:14.807529864 +0000 UTC m=+0.366328839,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-31.ec2.internal,}"
Jul 21 17:26:14.815158 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.815129    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 21 17:26:14.817020 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.817000    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 21 17:26:14.817020 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.817009    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 21 17:26:14.817167 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.817081    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:14.817874 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.817861    2579 factory.go:55] Registering systemd factory
Jul 21 17:26:14.817945 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.817910    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 21 17:26:14.817945 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.817916    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 21 17:26:14.818045 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.817966    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 21 17:26:14.818045 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.817978    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 21 17:26:14.818114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.818064    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 21 17:26:14.818114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.818074    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 21 17:26:14.818114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.818101    2579 factory.go:153] Registering CRI-O factory
Jul 21 17:26:14.818114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.818115    2579 factory.go:223] Registration of the crio container factory successfully
Jul 21 17:26:14.818214 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.818164    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 21 17:26:14.818214 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.818184    2579 factory.go:103] Registering Raw factory
Jul 21 17:26:14.818214 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.818197    2579 manager.go:1196] Started watching for new ooms in manager
Jul 21 17:26:14.819068 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.819054    2579 manager.go:349] Starting recovery of all containers
Jul 21 17:26:14.819405 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.819340    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 21 17:26:14.819498 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.819483    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 21 17:26:14.820304 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.820281    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvc4v"
Jul 21 17:26:14.826423 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.826293    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 21 17:26:14.828967 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.828946    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvc4v"
Jul 21 17:26:14.829691 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.829674    2579 manager.go:354] Recovery completed
Jul 21 17:26:14.836898 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.836882    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:14.838914 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.838899    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:14.838990 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.838932    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:14.838990 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.838948    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:14.839534 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.839518    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 21 17:26:14.839534 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.839532    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 21 17:26:14.839638 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.839541    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:14.839638 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.839547    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 21 17:26:14.840328 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.840301    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:14.840328 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.840328    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:14.840425 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.840338    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:14.840782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.840770    2579 policy_none.go:49] "None policy: Start"
Jul 21 17:26:14.840842 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.840785    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 21 17:26:14.840842 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.840795    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 21 17:26:14.841725 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.841712    2579 policy_none.go:47] "Start"
Jul 21 17:26:14.875301 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.875274    2579 manager.go:337] "Starting Device Plugin manager"
Jul 21 17:26:14.875441 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.875313    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 21 17:26:14.875441 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.875326    2579 server.go:87] "Starting device plugin registration server"
Jul 21 17:26:14.875640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.875626    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 21 17:26:14.875712 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.875643    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 21 17:26:14.875804 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.875763    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 21 17:26:14.875905 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.875882    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 21 17:26:14.875905 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.875892    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 21 17:26:14.876503 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.876482    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 21 17:26:14.876576 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.876541    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:14.976197 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.976165    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:14.977363 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.977115    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:14.977363 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.977151    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:14.977363 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.977164    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:14.977363 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.977192    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-31.ec2.internal"
Jul 21 17:26:14.986765 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.986715    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-31.ec2.internal"
Jul 21 17:26:14.986765 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.986740    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-31.ec2.internal\": node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:14.997772 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.997750    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 21 17:26:14.997889 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.997790    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 21 17:26:14.997889 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:14.997811    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 21 17:26:14.997889 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:14.997841    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 21 17:26:15.000917 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.000896    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:15.007022 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.007006    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:15.098164 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.098119    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal"]
Jul 21 17:26:15.098318 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.098212    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:15.099897 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.099881    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:15.099978 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.099911    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:15.099978 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.099922    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:15.101224 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.101212    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:15.101329 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.101316    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.101362 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.101358    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:15.101974 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.101957    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:15.102087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.101969    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:15.102087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.101987    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:15.102087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.102042    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:15.102087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.102007    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:15.102087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.102082    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:15.103145 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.103129    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.103215 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.103154    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:15.103890 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.103875    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:15.103958 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.103907    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:15.103958 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.103922    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:15.107117 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.107092    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:15.119638 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.119616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a79463808a4d0705f9376bbce08cef7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal\" (UID: \"0a79463808a4d0705f9376bbce08cef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.119781 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.119646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1faed2aff0ab49640344e2b4b87931f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-31.ec2.internal\" (UID: \"c1faed2aff0ab49640344e2b4b87931f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.119781 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.119684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a79463808a4d0705f9376bbce08cef7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal\" (UID: \"0a79463808a4d0705f9376bbce08cef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.131603 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.131578    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-31.ec2.internal\" not found" node="ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.136114 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.136094    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-31.ec2.internal\" not found" node="ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.207562 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.207530    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:15.220522 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.220496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a79463808a4d0705f9376bbce08cef7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal\" (UID: \"0a79463808a4d0705f9376bbce08cef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.220606 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.220531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1faed2aff0ab49640344e2b4b87931f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-31.ec2.internal\" (UID: \"c1faed2aff0ab49640344e2b4b87931f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.220606 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.220560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a79463808a4d0705f9376bbce08cef7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal\" (UID: \"0a79463808a4d0705f9376bbce08cef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.220692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.220623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a79463808a4d0705f9376bbce08cef7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal\" (UID: \"0a79463808a4d0705f9376bbce08cef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.220692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.220634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1faed2aff0ab49640344e2b4b87931f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-31.ec2.internal\" (UID: \"c1faed2aff0ab49640344e2b4b87931f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.220692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.220646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a79463808a4d0705f9376bbce08cef7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal\" (UID: \"0a79463808a4d0705f9376bbce08cef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.307828 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.307763    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:15.408372 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.408332    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:15.435900 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.435867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.439801 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.439582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:15.509417 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.509369    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:15.609965 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.609930    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:15.710447 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.710426    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:15.760021 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.759990    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 21 17:26:15.760533 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.760149    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 21 17:26:15.811491 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.811453    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:15.833344 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.833307    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 17:21:15 +0000 UTC" deadline="2028-04-01 08:40:46.185474628 +0000 UTC"
Jul 21 17:26:15.833344 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.833335    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14871h14m30.352143128s"
Jul 21 17:26:15.903174 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.903154    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:15.911777 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:15.911754    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:15.913356 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:15.913329    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1faed2aff0ab49640344e2b4b87931f.slice/crio-87bb9b551b67d5cf968b533f289b9694cd9db907a176a7b51291ffeda61b9437 WatchSource:0}: Error finding container 87bb9b551b67d5cf968b533f289b9694cd9db907a176a7b51291ffeda61b9437: Status 404 returned error can't find the container with id 87bb9b551b67d5cf968b533f289b9694cd9db907a176a7b51291ffeda61b9437
Jul 21 17:26:15.913590 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:15.913571    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a79463808a4d0705f9376bbce08cef7.slice/crio-180bcdc662295f2dd5ea08670dca75eb5011b76dbac43895e78ed094e417a665 WatchSource:0}: Error finding container 180bcdc662295f2dd5ea08670dca75eb5011b76dbac43895e78ed094e417a665: Status 404 returned error can't find the container with id 180bcdc662295f2dd5ea08670dca75eb5011b76dbac43895e78ed094e417a665
Jul 21 17:26:15.919424 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.919408    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:26:15.971948 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:15.971921    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:16.001721 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.001665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal" event={"ID":"c1faed2aff0ab49640344e2b4b87931f","Type":"ContainerStarted","Data":"87bb9b551b67d5cf968b533f289b9694cd9db907a176a7b51291ffeda61b9437"}
Jul 21 17:26:16.002549 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.002531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal" event={"ID":"0a79463808a4d0705f9376bbce08cef7","Type":"ContainerStarted","Data":"180bcdc662295f2dd5ea08670dca75eb5011b76dbac43895e78ed094e417a665"}
Jul 21 17:26:16.012710 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:16.012691    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:16.113256 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:16.113219    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-31.ec2.internal\" not found"
Jul 21 17:26:16.133806 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.133780    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:16.217439 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.217367    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:16.234281 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.234261    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 21 17:26:16.235284 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.235268    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal"
Jul 21 17:26:16.245867 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.245847    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 21 17:26:16.792737 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.792690    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:16.797894 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.797864    2579 apiserver.go:52] "Watching apiserver"
Jul 21 17:26:16.811169 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.811135    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 21 17:26:16.811669 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.811619    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-sckv6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal","openshift-multus/multus-additional-cni-plugins-67qsr","openshift-multus/network-metrics-daemon-f6xmv","openshift-network-operator/iptables-alerter-c5m62","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4","openshift-cluster-node-tuning-operator/tuned-7wqqh","openshift-dns/node-resolver-mz828","openshift-multus/multus-qgmkm","openshift-network-diagnostics/network-check-target-8vgqn","openshift-ovn-kubernetes/ovnkube-node-mr248","kube-system/global-pull-secret-syncer-l5mfk","kube-system/konnectivity-agent-p87zq","kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal"]
Jul 21 17:26:16.813249 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.813221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.815465 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.815438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:16.817605 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.817504    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 21 17:26:16.817738 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.817652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.819587 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.819566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:16.819828 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:16.819691    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:16.820859 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.820829    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:16.822473 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.822453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:16.824335 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.824309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.826513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.825956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:16.828005 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.827987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-host\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dad033f-a636-412a-81fc-3e7b683e14e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dad033f-a636-412a-81fc-3e7b683e14e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-sysctl-conf\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1-agent-certs\") pod \"konnectivity-agent-p87zq\" (UID: \"b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1\") " pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-run\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6mtp\" (UniqueName: \"kubernetes.io/projected/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-kube-api-access-m6mtp\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-sysconfig\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-sysctl-d\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-systemd\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a4707f9-96d8-41ac-9c10-09fa08d73e89-tmp\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-os-release\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-lib-modules\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-modprobe-d\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.830400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.829994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-kubernetes\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:16.829772    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.830017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-system-cni-dir\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.830056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-var-lib-kubelet\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.830158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1-konnectivity-ca\") pod \"konnectivity-agent-p87zq\" (UID: \"b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1\") " pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.830188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-cnibin\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.830213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dad033f-a636-412a-81fc-3e7b683e14e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.830292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.830327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-sys\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.830353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-tuned\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.830401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvmj\" (UniqueName: \"kubernetes.io/projected/0a4707f9-96d8-41ac-9c10-09fa08d73e89-kube-api-access-9rvmj\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.831543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.830428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dc6\" (UniqueName: \"kubernetes.io/projected/5dad033f-a636-412a-81fc-3e7b683e14e8-kube-api-access-h2dc6\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.832139 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.831999    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 21 17:26:16.832981 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.832561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.833371 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.833347    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 21 17:26:16.833588 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.833571    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:26:16.833690 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.833652    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 21 17:26:16.833690 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.833670    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b4nm4\""
Jul 21 17:26:16.833806 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.833765    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 21 17:26:16.834163 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.834140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:16.834270 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:16.834206    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:16.840495 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.840468    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 21 17:26:16.840647 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.840569    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 21 17:26:16.840723 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.840661    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 21 17:26:16.840723 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.840630    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 21 17:26:16.840723 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.840712    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 21 17:26:16.840854 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.840468    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 21 17:26:16.840854 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.840813    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:26:16.840981 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.840961    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 21 17:26:16.841145 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.841116    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.841830    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.841880    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842129    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842210    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842226    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842234    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rhcjl\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842245    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842226    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842298    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842366    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d8t8h\""
Jul 21 17:26:16.842456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842435    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 21 17:26:16.843084 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842529    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2qn7t\""
Jul 21 17:26:16.843084 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842633    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 21 17:26:16.843084 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.842652    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ts8mj\""
Jul 21 17:26:16.843239 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.843201    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zpkt4\""
Jul 21 17:26:16.843371 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.843348    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6qr87\""
Jul 21 17:26:16.845252 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.845229    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 21 17:26:16.845349 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.845284    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 21 17:26:16.846401 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.845676    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q5n92\""
Jul 21 17:26:16.846401 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.845991    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 21 17:26:16.846699 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.846666    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lb7v5\""
Jul 21 17:26:16.869627 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.869597    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 21 17:26:16.914358 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.914326    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6647q"
Jul 21 17:26:16.919508 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.919484    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 21 17:26:16.928763 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.928734    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6647q"
Jul 21 17:26:16.930913 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.930886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dkl\" (UniqueName: \"kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl\") pod \"network-check-target-8vgqn\" (UID: \"53d47a1a-f442-4cac-abd3-941e47645e03\") " pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:16.931039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.930930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.931039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.930961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74g7g\" (UniqueName: \"kubernetes.io/projected/6bbc7a18-e89b-4b13-b579-452215ed3b3e-kube-api-access-74g7g\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:16.931039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.930993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a4707f9-96d8-41ac-9c10-09fa08d73e89-tmp\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.931039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-cnibin\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.931291 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5mjg\" (UniqueName: \"kubernetes.io/projected/3b03ab87-1a4b-4b9d-b530-e3c8d082464d-kube-api-access-w5mjg\") pod \"node-resolver-mz828\" (UID: \"3b03ab87-1a4b-4b9d-b530-e3c8d082464d\") " pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:16.931291 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-modprobe-d\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.931291 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-kubernetes\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.931291 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-kubernetes\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.931291 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-modprobe-d\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.931291 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-system-cni-dir\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.931291 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-cni-bin\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.931291 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-registration-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:16.931685 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-var-lib-kubelet\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.931685 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931349    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 21 17:26:16.931685 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-system-cni-dir\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.931685 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1-konnectivity-ca\") pod \"konnectivity-agent-p87zq\" (UID: \"b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1\") " pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:16.931685 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-var-lib-kubelet\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.931685 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-run-k8s-cni-cncf-io\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.931997 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-etc-kubernetes\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.931997 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-cni-netd\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.931997 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af64258e-d822-4ebb-b7a4-9ec5a1b71267-ovnkube-script-lib\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.931997 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ck8z\" (UniqueName: \"kubernetes.io/projected/af64258e-d822-4ebb-b7a4-9ec5a1b71267-kube-api-access-9ck8z\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.931997 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-kubelet-config\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:16.931997 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-tuned\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.931997 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1-konnectivity-ca\") pod \"konnectivity-agent-p87zq\" (UID: \"b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1\") " pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:16.932331 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.931997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvmj\" (UniqueName: \"kubernetes.io/projected/0a4707f9-96d8-41ac-9c10-09fa08d73e89-kube-api-access-9rvmj\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.932331 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dc6\" (UniqueName: \"kubernetes.io/projected/5dad033f-a636-412a-81fc-3e7b683e14e8-kube-api-access-h2dc6\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.932331 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5856df40-592b-4d35-becf-d70792f8e5cf-cni-binary-copy\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.932331 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-hostroot\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.932331 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkhxk\" (UniqueName: \"kubernetes.io/projected/59fc317b-41ba-4e6e-8b7c-5ae9112a89cf-kube-api-access-bkhxk\") pod \"node-ca-sckv6\" (UID: \"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf\") " pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:16.932331 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-systemd-units\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.932331 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-dbus\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:16.932331 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-host\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-run-netns\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-slash\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:16.932428    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-host\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:16.932495    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs podName:a3f8ea96-7aec-48b2-85ef-6fe96c80ce62 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:17.432473641 +0000 UTC m=+2.991272628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs") pod "network-metrics-daemon-f6xmv" (UID: "a3f8ea96-7aec-48b2-85ef-6fe96c80ce62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-var-lib-openvswitch\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af64258e-d822-4ebb-b7a4-9ec5a1b71267-ovnkube-config\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-socket-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-var-lib-cni-multus\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6jq\" (UniqueName: \"kubernetes.io/projected/5856df40-592b-4d35-becf-d70792f8e5cf-kube-api-access-sb6jq\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-node-log\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-device-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:16.932782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1-agent-certs\") pod \"konnectivity-agent-p87zq\" (UID: \"b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1\") " pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-multus-cni-dir\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-var-lib-cni-bin\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5856df40-592b-4d35-becf-d70792f8e5cf-multus-daemon-config\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-run\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.932955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6mtp\" (UniqueName: \"kubernetes.io/projected/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-kube-api-access-m6mtp\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-run\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-sysconfig\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-sysctl-d\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-systemd\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-etc-openvswitch\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ghcn\" (UniqueName: \"kubernetes.io/projected/add844a8-7efd-4d4f-9dd3-535fbae49877-kube-api-access-6ghcn\") pod \"iptables-alerter-c5m62\" (UID: \"add844a8-7efd-4d4f-9dd3-535fbae49877\") " pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-os-release\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-systemd\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-os-release\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-sysconfig\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-sysctl-d\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.933502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-multus-socket-dir-parent\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59fc317b-41ba-4e6e-8b7c-5ae9112a89cf-host\") pod \"node-ca-sckv6\" (UID: \"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf\") " pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-run-ovn\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af64258e-d822-4ebb-b7a4-9ec5a1b71267-ovn-node-metrics-cert\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-lib-modules\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-run-openvswitch\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-cnibin\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dad033f-a636-412a-81fc-3e7b683e14e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-lib-modules\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-multus-conf-dir\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-run-multus-certs\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59fc317b-41ba-4e6e-8b7c-5ae9112a89cf-serviceca\") pod \"node-ca-sckv6\" (UID: \"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf\") " pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-run-systemd\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af64258e-d822-4ebb-b7a4-9ec5a1b71267-env-overrides\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-sys\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.934407 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-run-netns\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dad033f-a636-412a-81fc-3e7b683e14e8-cnibin\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.933955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dad033f-a636-412a-81fc-3e7b683e14e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.934525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dad033f-a636-412a-81fc-3e7b683e14e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.934563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dad033f-a636-412a-81fc-3e7b683e14e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.934571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-system-cni-dir\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.934639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-sys\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.934710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-run-ovn-kubernetes\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.934742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/add844a8-7efd-4d4f-9dd3-535fbae49877-iptables-alerter-script\") pod \"iptables-alerter-c5m62\" (UID: \"add844a8-7efd-4d4f-9dd3-535fbae49877\") " pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.934779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dad033f-a636-412a-81fc-3e7b683e14e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.934822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-kubelet\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.935275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a4707f9-96d8-41ac-9c10-09fa08d73e89-tmp\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dad033f-a636-412a-81fc-3e7b683e14e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/add844a8-7efd-4d4f-9dd3-535fbae49877-host-slash\") pod \"iptables-alerter-c5m62\" (UID: \"add844a8-7efd-4d4f-9dd3-535fbae49877\") " pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-sys-fs\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-sysctl-conf\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-var-lib-kubelet\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-log-socket\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b03ab87-1a4b-4b9d-b530-e3c8d082464d-hosts-file\") pod \"node-resolver-mz828\" (UID: \"3b03ab87-1a4b-4b9d-b530-e3c8d082464d\") " pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b03ab87-1a4b-4b9d-b530-e3c8d082464d-tmp-dir\") pod \"node-resolver-mz828\" (UID: \"3b03ab87-1a4b-4b9d-b530-e3c8d082464d\") " pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-sysctl-conf\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.935919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.935584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-os-release\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:16.936777 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.936753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a4707f9-96d8-41ac-9c10-09fa08d73e89-etc-tuned\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:16.938865 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.938800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1-agent-certs\") pod \"konnectivity-agent-p87zq\" (UID: \"b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1\") " pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:16.947884 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.947838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dc6\" (UniqueName: \"kubernetes.io/projected/5dad033f-a636-412a-81fc-3e7b683e14e8-kube-api-access-h2dc6\") pod \"multus-additional-cni-plugins-67qsr\" (UID: \"5dad033f-a636-412a-81fc-3e7b683e14e8\") " pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:16.959216 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.959191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6mtp\" (UniqueName: \"kubernetes.io/projected/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-kube-api-access-m6mtp\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:16.959216 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:16.959205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvmj\" (UniqueName: \"kubernetes.io/projected/0a4707f9-96d8-41ac-9c10-09fa08d73e89-kube-api-access-9rvmj\") pod \"tuned-7wqqh\" (UID: \"0a4707f9-96d8-41ac-9c10-09fa08d73e89\") " pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:17.036253 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-etc-openvswitch\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.036487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ghcn\" (UniqueName: \"kubernetes.io/projected/add844a8-7efd-4d4f-9dd3-535fbae49877-kube-api-access-6ghcn\") pod \"iptables-alerter-c5m62\" (UID: \"add844a8-7efd-4d4f-9dd3-535fbae49877\") " pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:17.036487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-multus-socket-dir-parent\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.036487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59fc317b-41ba-4e6e-8b7c-5ae9112a89cf-host\") pod \"node-ca-sckv6\" (UID: \"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf\") " pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:17.036487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-etc-openvswitch\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.036487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-run-ovn\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.036487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af64258e-d822-4ebb-b7a4-9ec5a1b71267-ovn-node-metrics-cert\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.036487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:17.036487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59fc317b-41ba-4e6e-8b7c-5ae9112a89cf-host\") pod \"node-ca-sckv6\" (UID: \"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf\") " pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:17.036487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-multus-socket-dir-parent\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.036487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-run-ovn\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.036495    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-run-openvswitch\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.036553    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret podName:1511f43e-7fd8-4f20-86ca-f9eb3c032ac5 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:17.536532764 +0000 UTC m=+3.095331745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret") pod "global-pull-secret-syncer-l5mfk" (UID: "1511f43e-7fd8-4f20-86ca-f9eb3c032ac5") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-run-openvswitch\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-multus-conf-dir\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-run-multus-certs\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59fc317b-41ba-4e6e-8b7c-5ae9112a89cf-serviceca\") pod \"node-ca-sckv6\" (UID: \"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf\") " pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-multus-conf-dir\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-run-systemd\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-run-multus-certs\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af64258e-d822-4ebb-b7a4-9ec5a1b71267-env-overrides\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-run-netns\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-run-netns\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-run-systemd\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-system-cni-dir\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.037175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-system-cni-dir\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-run-ovn-kubernetes\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/add844a8-7efd-4d4f-9dd3-535fbae49877-iptables-alerter-script\") pod \"iptables-alerter-c5m62\" (UID: \"add844a8-7efd-4d4f-9dd3-535fbae49877\") " pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.036995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-kubelet\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/add844a8-7efd-4d4f-9dd3-535fbae49877-host-slash\") pod \"iptables-alerter-c5m62\" (UID: \"add844a8-7efd-4d4f-9dd3-535fbae49877\") " pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-sys-fs\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-run-ovn-kubernetes\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-var-lib-kubelet\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-log-socket\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/add844a8-7efd-4d4f-9dd3-535fbae49877-host-slash\") pod \"iptables-alerter-c5m62\" (UID: \"add844a8-7efd-4d4f-9dd3-535fbae49877\") " pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59fc317b-41ba-4e6e-8b7c-5ae9112a89cf-serviceca\") pod \"node-ca-sckv6\" (UID: \"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf\") " pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-log-socket\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-var-lib-kubelet\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-kubelet\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-sys-fs\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.038106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af64258e-d822-4ebb-b7a4-9ec5a1b71267-env-overrides\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b03ab87-1a4b-4b9d-b530-e3c8d082464d-hosts-file\") pod \"node-resolver-mz828\" (UID: \"3b03ab87-1a4b-4b9d-b530-e3c8d082464d\") " pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b03ab87-1a4b-4b9d-b530-e3c8d082464d-tmp-dir\") pod \"node-resolver-mz828\" (UID: \"3b03ab87-1a4b-4b9d-b530-e3c8d082464d\") " pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-os-release\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b03ab87-1a4b-4b9d-b530-e3c8d082464d-hosts-file\") pod \"node-resolver-mz828\" (UID: \"3b03ab87-1a4b-4b9d-b530-e3c8d082464d\") " pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dkl\" (UniqueName: \"kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl\") pod \"network-check-target-8vgqn\" (UID: \"53d47a1a-f442-4cac-abd3-941e47645e03\") " pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74g7g\" (UniqueName: \"kubernetes.io/projected/6bbc7a18-e89b-4b13-b579-452215ed3b3e-kube-api-access-74g7g\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-os-release\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-cnibin\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5mjg\" (UniqueName: \"kubernetes.io/projected/3b03ab87-1a4b-4b9d-b530-e3c8d082464d-kube-api-access-w5mjg\") pod \"node-resolver-mz828\" (UID: \"3b03ab87-1a4b-4b9d-b530-e3c8d082464d\") " pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-cni-bin\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-registration-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-run-k8s-cni-cncf-io\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-etc-kubernetes\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.038996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b03ab87-1a4b-4b9d-b530-e3c8d082464d-tmp-dir\") pod \"node-resolver-mz828\" (UID: \"3b03ab87-1a4b-4b9d-b530-e3c8d082464d\") " pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-cni-netd\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/add844a8-7efd-4d4f-9dd3-535fbae49877-iptables-alerter-script\") pod \"iptables-alerter-c5m62\" (UID: \"add844a8-7efd-4d4f-9dd3-535fbae49877\") " pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-cni-netd\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-cni-bin\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-registration-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-run-k8s-cni-cncf-io\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-etc-kubernetes\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-cnibin\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af64258e-d822-4ebb-b7a4-9ec5a1b71267-ovnkube-script-lib\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ck8z\" (UniqueName: \"kubernetes.io/projected/af64258e-d822-4ebb-b7a4-9ec5a1b71267-kube-api-access-9ck8z\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-kubelet-config\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5856df40-592b-4d35-becf-d70792f8e5cf-cni-binary-copy\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.037994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-hostroot\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhxk\" (UniqueName: \"kubernetes.io/projected/59fc317b-41ba-4e6e-8b7c-5ae9112a89cf-kube-api-access-bkhxk\") pod \"node-ca-sckv6\" (UID: \"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf\") " pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-systemd-units\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-dbus\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-kubelet-config\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:17.039640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-run-netns\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-slash\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-var-lib-openvswitch\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af64258e-d822-4ebb-b7a4-9ec5a1b71267-ovnkube-config\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-dbus\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-socket-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-var-lib-cni-multus\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6jq\" (UniqueName: \"kubernetes.io/projected/5856df40-592b-4d35-becf-d70792f8e5cf-kube-api-access-sb6jq\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-node-log\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-device-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-multus-cni-dir\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-var-lib-cni-bin\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af64258e-d822-4ebb-b7a4-9ec5a1b71267-ovnkube-script-lib\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5856df40-592b-4d35-becf-d70792f8e5cf-multus-daemon-config\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-hostroot\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-systemd-units\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-socket-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-run-netns\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-var-lib-cni-multus\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-host-slash\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.040904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-var-lib-openvswitch\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.040904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6bbc7a18-e89b-4b13-b579-452215ed3b3e-device-dir\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.040904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-host-var-lib-cni-bin\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af64258e-d822-4ebb-b7a4-9ec5a1b71267-node-log\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.040904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5856df40-592b-4d35-becf-d70792f8e5cf-cni-binary-copy\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.039274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af64258e-d822-4ebb-b7a4-9ec5a1b71267-ovn-node-metrics-cert\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.040904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.038727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5856df40-592b-4d35-becf-d70792f8e5cf-multus-cni-dir\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.039573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5856df40-592b-4d35-becf-d70792f8e5cf-multus-daemon-config\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.040904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.039682    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af64258e-d822-4ebb-b7a4-9ec5a1b71267-ovnkube-config\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.047399 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.047333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ghcn\" (UniqueName: \"kubernetes.io/projected/add844a8-7efd-4d4f-9dd3-535fbae49877-kube-api-access-6ghcn\") pod \"iptables-alerter-c5m62\" (UID: \"add844a8-7efd-4d4f-9dd3-535fbae49877\") " pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:17.051623 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.051593    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:17.051623 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.051621    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:17.051816 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.051636    2579 projected.go:196] Error preparing data for projected volume kube-api-access-w6dkl for pod openshift-network-diagnostics/network-check-target-8vgqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:17.051816 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.051736    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl podName:53d47a1a-f442-4cac-abd3-941e47645e03 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:17.551717089 +0000 UTC m=+3.110516079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w6dkl" (UniqueName: "kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl") pod "network-check-target-8vgqn" (UID: "53d47a1a-f442-4cac-abd3-941e47645e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:17.053149 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.053019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5mjg\" (UniqueName: \"kubernetes.io/projected/3b03ab87-1a4b-4b9d-b530-e3c8d082464d-kube-api-access-w5mjg\") pod \"node-resolver-mz828\" (UID: \"3b03ab87-1a4b-4b9d-b530-e3c8d082464d\") " pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:17.053278 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.053259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhxk\" (UniqueName: \"kubernetes.io/projected/59fc317b-41ba-4e6e-8b7c-5ae9112a89cf-kube-api-access-bkhxk\") pod \"node-ca-sckv6\" (UID: \"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf\") " pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:17.053326 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.053261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ck8z\" (UniqueName: \"kubernetes.io/projected/af64258e-d822-4ebb-b7a4-9ec5a1b71267-kube-api-access-9ck8z\") pod \"ovnkube-node-mr248\" (UID: \"af64258e-d822-4ebb-b7a4-9ec5a1b71267\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.053591 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.053567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74g7g\" (UniqueName: \"kubernetes.io/projected/6bbc7a18-e89b-4b13-b579-452215ed3b3e-kube-api-access-74g7g\") pod \"aws-ebs-csi-driver-node-stdd4\" (UID: \"6bbc7a18-e89b-4b13-b579-452215ed3b3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.054633 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.054610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6jq\" (UniqueName: \"kubernetes.io/projected/5856df40-592b-4d35-becf-d70792f8e5cf-kube-api-access-sb6jq\") pod \"multus-qgmkm\" (UID: \"5856df40-592b-4d35-becf-d70792f8e5cf\") " pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.127215 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.127185    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7wqqh"
Jul 21 17:26:17.134931 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.134905    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:17.150003 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.149978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-67qsr"
Jul 21 17:26:17.157310 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.157285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c5m62"
Jul 21 17:26:17.165008 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.164986    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4"
Jul 21 17:26:17.173032 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.173006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qgmkm"
Jul 21 17:26:17.181028 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.180996    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sckv6"
Jul 21 17:26:17.189895 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.189877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mz828"
Jul 21 17:26:17.196986 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.196969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:17.442103 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.442071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:17.442290 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.442171    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:17.442290 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.442220    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs podName:a3f8ea96-7aec-48b2-85ef-6fe96c80ce62 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:18.442206128 +0000 UTC m=+4.001005092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs") pod "network-metrics-daemon-f6xmv" (UID: "a3f8ea96-7aec-48b2-85ef-6fe96c80ce62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:17.542421 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.542383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:17.542617 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.542509    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:17.542617 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.542589    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret podName:1511f43e-7fd8-4f20-86ca-f9eb3c032ac5 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:18.542567255 +0000 UTC m=+4.101366236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret") pod "global-pull-secret-syncer-l5mfk" (UID: "1511f43e-7fd8-4f20-86ca-f9eb3c032ac5") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:17.643860 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.643802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dkl\" (UniqueName: \"kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl\") pod \"network-check-target-8vgqn\" (UID: \"53d47a1a-f442-4cac-abd3-941e47645e03\") " pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:17.644070 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.644050    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:17.644131 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.644082    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:17.644131 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.644096    2579 projected.go:196] Error preparing data for projected volume kube-api-access-w6dkl for pod openshift-network-diagnostics/network-check-target-8vgqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:17.644211 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.644160    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl podName:53d47a1a-f442-4cac-abd3-941e47645e03 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:18.64413767 +0000 UTC m=+4.202936639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6dkl" (UniqueName: "kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl") pod "network-check-target-8vgqn" (UID: "53d47a1a-f442-4cac-abd3-941e47645e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:17.862913 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:17.862879    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b03ab87_1a4b_4b9d_b530_e3c8d082464d.slice/crio-fe635f4c4c18afb9f691ffb0159dbea2a96b16e9bb084fa8d8948fa73f48d28b WatchSource:0}: Error finding container fe635f4c4c18afb9f691ffb0159dbea2a96b16e9bb084fa8d8948fa73f48d28b: Status 404 returned error can't find the container with id fe635f4c4c18afb9f691ffb0159dbea2a96b16e9bb084fa8d8948fa73f48d28b
Jul 21 17:26:17.863604 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:17.863578    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4707f9_96d8_41ac_9c10_09fa08d73e89.slice/crio-132db1a549c133fd77cb054047229f6e1f1b50c6618868d99f42fc3cd4c3fef3 WatchSource:0}: Error finding container 132db1a549c133fd77cb054047229f6e1f1b50c6618868d99f42fc3cd4c3fef3: Status 404 returned error can't find the container with id 132db1a549c133fd77cb054047229f6e1f1b50c6618868d99f42fc3cd4c3fef3
Jul 21 17:26:17.868743 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:17.868717    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf64258e_d822_4ebb_b7a4_9ec5a1b71267.slice/crio-f3794f1076def1d489cf96daf4df36010e600acffa4572ee11e346f16c65da13 WatchSource:0}: Error finding container f3794f1076def1d489cf96daf4df36010e600acffa4572ee11e346f16c65da13: Status 404 returned error can't find the container with id f3794f1076def1d489cf96daf4df36010e600acffa4572ee11e346f16c65da13
Jul 21 17:26:17.869963 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:17.869636    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5856df40_592b_4d35_becf_d70792f8e5cf.slice/crio-9b6e52a2670a57789c4db2e768ce7b960164c25a12e43f3c7da7db294f474d3d WatchSource:0}: Error finding container 9b6e52a2670a57789c4db2e768ce7b960164c25a12e43f3c7da7db294f474d3d: Status 404 returned error can't find the container with id 9b6e52a2670a57789c4db2e768ce7b960164c25a12e43f3c7da7db294f474d3d
Jul 21 17:26:17.870573 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:17.870548    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbc7a18_e89b_4b13_b579_452215ed3b3e.slice/crio-30df3103b18e80908bf2372b5291a750df72acbfcad53d23307394fc3ea73425 WatchSource:0}: Error finding container 30df3103b18e80908bf2372b5291a750df72acbfcad53d23307394fc3ea73425: Status 404 returned error can't find the container with id 30df3103b18e80908bf2372b5291a750df72acbfcad53d23307394fc3ea73425
Jul 21 17:26:17.931146 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.931001    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 17:21:17 +0000 UTC" deadline="2028-01-28 05:49:15.909654994 +0000 UTC"
Jul 21 17:26:17.931146 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.931143    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13332h22m57.978516192s"
Jul 21 17:26:17.998205 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:17.998174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:17.998370 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:17.998286    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:18.007056 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.007027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-67qsr" event={"ID":"5dad033f-a636-412a-81fc-3e7b683e14e8","Type":"ContainerStarted","Data":"334cac41d6f4af981be1ec691ce608970f06fcdbbc2d9c5b95d738d9c7640831"}
Jul 21 17:26:18.008383 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.008355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c5m62" event={"ID":"add844a8-7efd-4d4f-9dd3-535fbae49877","Type":"ContainerStarted","Data":"7710aece8415308aac3e328eda1549934b4e8df6733549e5bd30949b39ab205a"}
Jul 21 17:26:18.009554 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.009531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sckv6" event={"ID":"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf","Type":"ContainerStarted","Data":"26ac7e9ecb100800fb827a576b3324bd27e04f6673c0ba68be5d8a8c04a9c8ad"}
Jul 21 17:26:18.010857 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.010835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7wqqh" event={"ID":"0a4707f9-96d8-41ac-9c10-09fa08d73e89","Type":"ContainerStarted","Data":"132db1a549c133fd77cb054047229f6e1f1b50c6618868d99f42fc3cd4c3fef3"}
Jul 21 17:26:18.011982 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.011963    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mz828" event={"ID":"3b03ab87-1a4b-4b9d-b530-e3c8d082464d","Type":"ContainerStarted","Data":"fe635f4c4c18afb9f691ffb0159dbea2a96b16e9bb084fa8d8948fa73f48d28b"}
Jul 21 17:26:18.014090 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.014068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal" event={"ID":"c1faed2aff0ab49640344e2b4b87931f","Type":"ContainerStarted","Data":"4b0239263a6c8fdd231e4f28f19a327c56873adcf64e581ba0f31ab6fa1366e5"}
Jul 21 17:26:18.015366 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.015342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p87zq" event={"ID":"b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1","Type":"ContainerStarted","Data":"6f24d32d8e5d3c1897606619dfbd21032a371e980fa02d11347b7da2dd8e9026"}
Jul 21 17:26:18.016613 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.016594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4" event={"ID":"6bbc7a18-e89b-4b13-b579-452215ed3b3e","Type":"ContainerStarted","Data":"30df3103b18e80908bf2372b5291a750df72acbfcad53d23307394fc3ea73425"}
Jul 21 17:26:18.017884 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.017858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qgmkm" event={"ID":"5856df40-592b-4d35-becf-d70792f8e5cf","Type":"ContainerStarted","Data":"9b6e52a2670a57789c4db2e768ce7b960164c25a12e43f3c7da7db294f474d3d"}
Jul 21 17:26:18.021805 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.021770    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr248" event={"ID":"af64258e-d822-4ebb-b7a4-9ec5a1b71267","Type":"ContainerStarted","Data":"f3794f1076def1d489cf96daf4df36010e600acffa4572ee11e346f16c65da13"}
Jul 21 17:26:18.451788 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.451722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:18.451992 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:18.451950    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:18.452062 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:18.452019    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs podName:a3f8ea96-7aec-48b2-85ef-6fe96c80ce62 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:20.451997944 +0000 UTC m=+6.010796911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs") pod "network-metrics-daemon-f6xmv" (UID: "a3f8ea96-7aec-48b2-85ef-6fe96c80ce62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:18.553095 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.552459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:18.553095 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:18.552637    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:18.553095 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:18.552719    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret podName:1511f43e-7fd8-4f20-86ca-f9eb3c032ac5 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:20.552699754 +0000 UTC m=+6.111498733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret") pod "global-pull-secret-syncer-l5mfk" (UID: "1511f43e-7fd8-4f20-86ca-f9eb3c032ac5") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:18.653100 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.653058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dkl\" (UniqueName: \"kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl\") pod \"network-check-target-8vgqn\" (UID: \"53d47a1a-f442-4cac-abd3-941e47645e03\") " pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:18.653375 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:18.653256    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:18.653375 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:18.653279    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:18.653375 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:18.653292    2579 projected.go:196] Error preparing data for projected volume kube-api-access-w6dkl for pod openshift-network-diagnostics/network-check-target-8vgqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:18.653375 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:18.653349    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl podName:53d47a1a-f442-4cac-abd3-941e47645e03 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:20.653331139 +0000 UTC m=+6.212130119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6dkl" (UniqueName: "kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl") pod "network-check-target-8vgqn" (UID: "53d47a1a-f442-4cac-abd3-941e47645e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:18.932004 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.931912    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 17:21:17 +0000 UTC" deadline="2027-12-20 14:24:42.574688074 +0000 UTC"
Jul 21 17:26:18.932004 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:18.931997    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12404h58m23.642696498s"
Jul 21 17:26:19.001545 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:19.001001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:19.001545 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:19.001124    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:19.002135 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:19.001777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:19.002135 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:19.001892    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:19.037698 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:19.036510    2579 generic.go:356] "Generic (PLEG): container finished" podID="0a79463808a4d0705f9376bbce08cef7" containerID="17c3f29073ee93ab73bbca0ac8d3554635b7ea97ce63a0a70138b4daa043d622" exitCode=0
Jul 21 17:26:19.037698 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:19.037463    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal" event={"ID":"0a79463808a4d0705f9376bbce08cef7","Type":"ContainerDied","Data":"17c3f29073ee93ab73bbca0ac8d3554635b7ea97ce63a0a70138b4daa043d622"}
Jul 21 17:26:19.062104 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:19.061715    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-31.ec2.internal" podStartSLOduration=2.061690196 podStartE2EDuration="2.061690196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:18.034001803 +0000 UTC m=+3.592800790" watchObservedRunningTime="2026-07-21 17:26:19.061690196 +0000 UTC m=+4.620489184"
Jul 21 17:26:19.999386 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:19.998838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:19.999386 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:19.999010    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:20.049030 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:20.047988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal" event={"ID":"0a79463808a4d0705f9376bbce08cef7","Type":"ContainerStarted","Data":"2067b730d51931f353b7c7e9c8817b42ecdc7e5f4bd8ae58ea10db14ab64b74f"}
Jul 21 17:26:20.471203 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:20.470405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:20.471203 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:20.470563    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:20.471203 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:20.470660    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs podName:a3f8ea96-7aec-48b2-85ef-6fe96c80ce62 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:24.470640637 +0000 UTC m=+10.029439620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs") pod "network-metrics-daemon-f6xmv" (UID: "a3f8ea96-7aec-48b2-85ef-6fe96c80ce62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:20.571819 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:20.571180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:20.571819 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:20.571381    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:20.571819 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:20.571447    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret podName:1511f43e-7fd8-4f20-86ca-f9eb3c032ac5 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:24.571429712 +0000 UTC m=+10.130228681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret") pod "global-pull-secret-syncer-l5mfk" (UID: "1511f43e-7fd8-4f20-86ca-f9eb3c032ac5") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:20.672430 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:20.671641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dkl\" (UniqueName: \"kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl\") pod \"network-check-target-8vgqn\" (UID: \"53d47a1a-f442-4cac-abd3-941e47645e03\") " pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:20.672430 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:20.671954    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:20.672430 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:20.671977    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:20.672430 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:20.671990    2579 projected.go:196] Error preparing data for projected volume kube-api-access-w6dkl for pod openshift-network-diagnostics/network-check-target-8vgqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:20.672430 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:20.672054    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl podName:53d47a1a-f442-4cac-abd3-941e47645e03 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:24.672033767 +0000 UTC m=+10.230832745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6dkl" (UniqueName: "kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl") pod "network-check-target-8vgqn" (UID: "53d47a1a-f442-4cac-abd3-941e47645e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:21.005487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:21.004652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:21.005487 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:21.004797    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:21.005487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:21.005186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:21.005487 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:21.005302    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:21.998488 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:21.998449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:21.998739 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:21.998592    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:22.998575 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:22.998536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:22.999097 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:22.998707    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:22.999185 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:22.999139    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:22.999331 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:22.999250    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:23.998408 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:23.998366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:23.998587 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:23.998512    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:24.507403 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:24.507301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:24.507951 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:24.507566    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:24.507951 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:24.507648    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs podName:a3f8ea96-7aec-48b2-85ef-6fe96c80ce62 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:32.507624399 +0000 UTC m=+18.066423376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs") pod "network-metrics-daemon-f6xmv" (UID: "a3f8ea96-7aec-48b2-85ef-6fe96c80ce62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:24.608612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:24.608573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:24.608819 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:24.608722    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:24.608819 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:24.608809    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret podName:1511f43e-7fd8-4f20-86ca-f9eb3c032ac5 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:32.608786692 +0000 UTC m=+18.167585655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret") pod "global-pull-secret-syncer-l5mfk" (UID: "1511f43e-7fd8-4f20-86ca-f9eb3c032ac5") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:24.710352 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:24.709632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dkl\" (UniqueName: \"kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl\") pod \"network-check-target-8vgqn\" (UID: \"53d47a1a-f442-4cac-abd3-941e47645e03\") " pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:24.710352 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:24.709881    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:24.710352 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:24.709922    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:24.710352 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:24.709936    2579 projected.go:196] Error preparing data for projected volume kube-api-access-w6dkl for pod openshift-network-diagnostics/network-check-target-8vgqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:24.710352 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:24.710004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl podName:53d47a1a-f442-4cac-abd3-941e47645e03 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:32.709984632 +0000 UTC m=+18.268783615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6dkl" (UniqueName: "kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl") pod "network-check-target-8vgqn" (UID: "53d47a1a-f442-4cac-abd3-941e47645e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:24.998932 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:24.998619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:24.998932 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:24.998785    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:25.000404 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:25.000267    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:25.000404 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:25.000357    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:25.998866 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:25.998827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:25.999364 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:25.998986    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:26.998343 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:26.998296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:26.998531 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:26.998296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:26.998531 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:26.998427    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:26.998796 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:26.998535    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:27.998034 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:27.998007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:27.998480 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:27.998124    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:28.998508 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:28.998473    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:28.998957 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:28.998616    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:28.998957 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:28.998623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:28.998957 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:28.998761    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:29.998361 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:29.998332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:29.998538 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:29.998438    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:30.999051 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:30.999012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:30.999513 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:30.999153    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:30.999513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:30.999211    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:30.999513 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:30.999331    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:31.998356 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:31.998321    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:31.998612 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:31.998455    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:32.570219 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:32.570191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:32.570689 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:32.570315    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:32.570689 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:32.570388    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs podName:a3f8ea96-7aec-48b2-85ef-6fe96c80ce62 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:48.5703654 +0000 UTC m=+34.129164363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs") pod "network-metrics-daemon-f6xmv" (UID: "a3f8ea96-7aec-48b2-85ef-6fe96c80ce62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:32.671015 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:32.670979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:32.671189 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:32.671098    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:32.671189 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:32.671154    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret podName:1511f43e-7fd8-4f20-86ca-f9eb3c032ac5 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:48.671139797 +0000 UTC m=+34.229938758 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret") pod "global-pull-secret-syncer-l5mfk" (UID: "1511f43e-7fd8-4f20-86ca-f9eb3c032ac5") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:32.771775 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:32.771733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dkl\" (UniqueName: \"kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl\") pod \"network-check-target-8vgqn\" (UID: \"53d47a1a-f442-4cac-abd3-941e47645e03\") " pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:32.771936 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:32.771880    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:32.771936 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:32.771900    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:32.771936 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:32.771911    2579 projected.go:196] Error preparing data for projected volume kube-api-access-w6dkl for pod openshift-network-diagnostics/network-check-target-8vgqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:32.772099 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:32.771971    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl podName:53d47a1a-f442-4cac-abd3-941e47645e03 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:48.771954136 +0000 UTC m=+34.330753111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6dkl" (UniqueName: "kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl") pod "network-check-target-8vgqn" (UID: "53d47a1a-f442-4cac-abd3-941e47645e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:32.998630 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:32.998593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:32.998821 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:32.998662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:32.998821 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:32.998739    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:32.998936 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:32.998883    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:33.998446 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:33.998412    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:33.998896 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:33.998542    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:34.998941 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:34.998906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:34.999348 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:34.998986    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:34.999348 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:34.999017    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:34.999348 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:34.999126    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:35.998712 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:35.998679    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:35.998891 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:35.998800    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:36.998495 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:36.998461    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:36.998973 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:36.998467    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:36.998973 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:36.998599    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:36.998973 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:36.998756    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:37.999044 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:37.998770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:37.999574 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:37.999119    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:38.082688 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.082627    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dad033f-a636-412a-81fc-3e7b683e14e8" containerID="e405bcb81806b52ceb9012498036460bbf6cbc24855dd931f2ae9f8540a49bde" exitCode=0
Jul 21 17:26:38.082771 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.082715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-67qsr" event={"ID":"5dad033f-a636-412a-81fc-3e7b683e14e8","Type":"ContainerDied","Data":"e405bcb81806b52ceb9012498036460bbf6cbc24855dd931f2ae9f8540a49bde"}
Jul 21 17:26:38.084172 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.084153    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sckv6" event={"ID":"59fc317b-41ba-4e6e-8b7c-5ae9112a89cf","Type":"ContainerStarted","Data":"15a6c376308752649d353572f43019382d89a7376e8171ed20bad0078a965823"}
Jul 21 17:26:38.085714 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.085692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7wqqh" event={"ID":"0a4707f9-96d8-41ac-9c10-09fa08d73e89","Type":"ContainerStarted","Data":"345d886c0affe54d2e74f10c111d12061388fbf6e40bb10b7085e8f80371725d"}
Jul 21 17:26:38.087145 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.087121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mz828" event={"ID":"3b03ab87-1a4b-4b9d-b530-e3c8d082464d","Type":"ContainerStarted","Data":"9f3c3de3cb3ad835421fc0f6ae712cae5e8f4eea619eb8f44e8de8ffba2b8246"}
Jul 21 17:26:38.088578 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.088558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p87zq" event={"ID":"b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1","Type":"ContainerStarted","Data":"6602ef2d3483e4a2bf67028da50509f1929fb3ecc6f4143020b53e2a8e4856f0"}
Jul 21 17:26:38.090843 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.090807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4" event={"ID":"6bbc7a18-e89b-4b13-b579-452215ed3b3e","Type":"ContainerStarted","Data":"0c7e21f465975470918b928853ad036da86bfdacce901e986ed5ab8f1cc80e2b"}
Jul 21 17:26:38.092279 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.092260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qgmkm" event={"ID":"5856df40-592b-4d35-becf-d70792f8e5cf","Type":"ContainerStarted","Data":"d845372538bfe07bfffe2877f7ff8d970ab4184609bd7ea32dc97d5c76e52161"}
Jul 21 17:26:38.095138 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.095120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr248" event={"ID":"af64258e-d822-4ebb-b7a4-9ec5a1b71267","Type":"ContainerStarted","Data":"7a7a03f973d179dd2661172e52bd3e95c16e9237439d25b06db40c46d34ac152"}
Jul 21 17:26:38.095231 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.095185    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr248" event={"ID":"af64258e-d822-4ebb-b7a4-9ec5a1b71267","Type":"ContainerStarted","Data":"f64c82c6e1d106c617ecf4c742f83fb9bd2347b85687e98d72e0b2577c1d4fed"}
Jul 21 17:26:38.095292 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.095239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr248" event={"ID":"af64258e-d822-4ebb-b7a4-9ec5a1b71267","Type":"ContainerStarted","Data":"a70f4dd151550210b13827a00a9a9fab173628fef5f2508a28688670c806ccbc"}
Jul 21 17:26:38.095292 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.095289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr248" event={"ID":"af64258e-d822-4ebb-b7a4-9ec5a1b71267","Type":"ContainerStarted","Data":"b5efdb7af8dfde632f58934fa4edd5d27f370ec6d122dfa74dee8b8eab63f679"}
Jul 21 17:26:38.095391 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.095336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr248" event={"ID":"af64258e-d822-4ebb-b7a4-9ec5a1b71267","Type":"ContainerStarted","Data":"c1afae73e9c69ae287bb121a9dd102b1f90b94487e23a8b74ec93e56e83e02b0"}
Jul 21 17:26:38.121026 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.120980    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-31.ec2.internal" podStartSLOduration=21.12095169 podStartE2EDuration="21.12095169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:20.070076505 +0000 UTC m=+5.628875495" watchObservedRunningTime="2026-07-21 17:26:38.12095169 +0000 UTC m=+23.679750675"
Jul 21 17:26:38.159863 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.159820    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qgmkm" podStartSLOduration=3.753866723 podStartE2EDuration="23.159805932s" totalImagesPullingTime="19.405939209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:38.159400933 +0000 UTC m=+23.718199929" watchObservedRunningTime="2026-07-21 17:26:38.159805932 +0000 UTC m=+23.718604915"
Jul 21 17:26:38.187754 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.187713    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mz828" podStartSLOduration=3.840725995 podStartE2EDuration="23.187698519s" totalImagesPullingTime="19.346972524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:38.187119646 +0000 UTC m=+23.745918630" watchObservedRunningTime="2026-07-21 17:26:38.187698519 +0000 UTC m=+23.746497502"
Jul 21 17:26:38.252546 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.252497    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sckv6" podStartSLOduration=3.917181276 podStartE2EDuration="23.252481311s" totalImagesPullingTime="19.335300035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:38.252240619 +0000 UTC m=+23.811039604" watchObservedRunningTime="2026-07-21 17:26:38.252481311 +0000 UTC m=+23.811280295"
Jul 21 17:26:38.252888 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.252860    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7wqqh" podStartSLOduration=3.903282291 podStartE2EDuration="23.252848542s" totalImagesPullingTime="19.349566251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:38.220779457 +0000 UTC m=+23.779578443" watchObservedRunningTime="2026-07-21 17:26:38.252848542 +0000 UTC m=+23.811647526"
Jul 21 17:26:38.277328 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.277291    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p87zq" podStartSLOduration=7.6776444040000005 podStartE2EDuration="23.2772784s" totalImagesPullingTime="15.599633996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:38.276815916 +0000 UTC m=+23.835614900" watchObservedRunningTime="2026-07-21 17:26:38.2772784 +0000 UTC m=+23.836077383"
Jul 21 17:26:38.604709 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.604674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:38.605376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.605355    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:38.801344 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.801323    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 21 17:26:38.891505 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.891341    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T17:26:38.801340987Z","UUID":"a3c7cee3-5555-4016-bf5d-149826dd3f6d","Handler":null,"Name":"","Endpoint":""}
Jul 21 17:26:38.892719 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.892702    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 21 17:26:38.892814 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.892737    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 21 17:26:38.998727 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.998701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:38.998896 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:38.998735    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:38.998896 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:38.998802    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:38.999001 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:38.998942    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:39.098450 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:39.098418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4" event={"ID":"6bbc7a18-e89b-4b13-b579-452215ed3b3e","Type":"ContainerStarted","Data":"a3dd6f4f05ce7ac600333281f8ac2b7507736b48ac80624274a2636fa14f69d5"}
Jul 21 17:26:39.101275 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:39.101251    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr248" event={"ID":"af64258e-d822-4ebb-b7a4-9ec5a1b71267","Type":"ContainerStarted","Data":"eeae5e78903cb64a4486ce8d47eaefaef2835d293bad9c32e216862d99fa1169"}
Jul 21 17:26:39.102492 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:39.102473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c5m62" event={"ID":"add844a8-7efd-4d4f-9dd3-535fbae49877","Type":"ContainerStarted","Data":"f111c3b1bc30ff5095fc6ee389ea0d14be5c7fbd1028470088510f05147c94f3"}
Jul 21 17:26:39.103035 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:39.103018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:39.103432 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:39.103417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p87zq"
Jul 21 17:26:39.125046 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:39.125010    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c5m62" podStartSLOduration=8.523192026 podStartE2EDuration="24.12499789s" totalImagesPullingTime="15.601805864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:39.124522825 +0000 UTC m=+24.683321808" watchObservedRunningTime="2026-07-21 17:26:39.12499789 +0000 UTC m=+24.683796873"
Jul 21 17:26:39.998551 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:39.998523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:39.998767 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:39.998630    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:40.107790 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:40.107759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4" event={"ID":"6bbc7a18-e89b-4b13-b579-452215ed3b3e","Type":"ContainerStarted","Data":"e9ca6fa6bb435ee29df3b9abbb82291d75a952ed3e50e65af82a2fddeff61a02"}
Jul 21 17:26:40.998344 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:40.998105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:40.998522 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:40.998105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:40.998522 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:40.998471    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:40.998635 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:40.998532    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:41.113683 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:41.113636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr248" event={"ID":"af64258e-d822-4ebb-b7a4-9ec5a1b71267","Type":"ContainerStarted","Data":"47cb1692c5bdfc19db2f461783a3e6e81cd9513b88b82d1f1a036696765441c3"}
Jul 21 17:26:41.998787 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:41.998759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:41.998964 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:41.998886    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:42.998431 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:42.998281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:42.998893 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:42.998333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:42.998893 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:42.998532    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:42.998893 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:42.998590    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:43.122123 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:43.122090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr248" event={"ID":"af64258e-d822-4ebb-b7a4-9ec5a1b71267","Type":"ContainerStarted","Data":"efb47e0e6538afdbe0891dfe77ef94c64e002e9e9fb4c291d372bf4f47458685"}
Jul 21 17:26:43.122371 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:43.122348    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:43.123914 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:43.123893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-67qsr" event={"ID":"5dad033f-a636-412a-81fc-3e7b683e14e8","Type":"ContainerStarted","Data":"f243fdaa5c253d7f61b46fee95302c0c45f7f886e58571c9403acf08b28c5799"}
Jul 21 17:26:43.137233 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:43.137209    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:43.164932 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:43.164885    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mr248" podStartSLOduration=8.747975022 podStartE2EDuration="28.164868847s" totalImagesPullingTime="19.416893825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:43.164840267 +0000 UTC m=+28.723639248" watchObservedRunningTime="2026-07-21 17:26:43.164868847 +0000 UTC m=+28.723667825"
Jul 21 17:26:43.165356 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:43.165334    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stdd4" podStartSLOduration=6.586213593 podStartE2EDuration="28.165325041s" totalImagesPullingTime="21.579111448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:40.135964233 +0000 UTC m=+25.694763214" watchObservedRunningTime="2026-07-21 17:26:43.165325041 +0000 UTC m=+28.724124059"
Jul 21 17:26:44.000000 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:43.998487    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:44.000000 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:43.998605    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:44.127232 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:44.127199    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dad033f-a636-412a-81fc-3e7b683e14e8" containerID="f243fdaa5c253d7f61b46fee95302c0c45f7f886e58571c9403acf08b28c5799" exitCode=0
Jul 21 17:26:44.127434 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:44.127284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-67qsr" event={"ID":"5dad033f-a636-412a-81fc-3e7b683e14e8","Type":"ContainerDied","Data":"f243fdaa5c253d7f61b46fee95302c0c45f7f886e58571c9403acf08b28c5799"}
Jul 21 17:26:44.127434 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:44.127342    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 21 17:26:44.127772 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:44.127754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:44.142856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:44.142834    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:44.571494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:44.571459    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:26:45.000279 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:45.000234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:45.000697 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:45.000353    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:45.000697 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:45.000396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:45.000697 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:45.000609    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:45.100487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:45.100281    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f6xmv"]
Jul 21 17:26:45.102752 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:45.102681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8vgqn"]
Jul 21 17:26:45.103449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:45.103426    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l5mfk"]
Jul 21 17:26:45.103547 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:45.103535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:45.103699 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:45.103671    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:45.132013 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:45.131986    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dad033f-a636-412a-81fc-3e7b683e14e8" containerID="abf8d9405dbe4c40ee9075bee4042cf347f7ce916f9ff77d1517619bf03bced4" exitCode=0
Jul 21 17:26:45.132155 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:45.132094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-67qsr" event={"ID":"5dad033f-a636-412a-81fc-3e7b683e14e8","Type":"ContainerDied","Data":"abf8d9405dbe4c40ee9075bee4042cf347f7ce916f9ff77d1517619bf03bced4"}
Jul 21 17:26:45.132155 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:45.132122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:45.132252 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:45.132232    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:45.132765 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:45.132678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:45.132892 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:45.132788    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:46.998406 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:46.998379    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:46.998958 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:46.998382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:46.998958 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:46.998527    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:46.998958 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:46.998382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:46.998958 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:46.998558    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:46.998958 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:46.998614    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:47.138683 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:47.138591    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dad033f-a636-412a-81fc-3e7b683e14e8" containerID="87ac0e02d8dc3994333097c268ba2a5dde78308150864e86cb5cb1c279d33aae" exitCode=0
Jul 21 17:26:47.138683 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:47.138633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-67qsr" event={"ID":"5dad033f-a636-412a-81fc-3e7b683e14e8","Type":"ContainerDied","Data":"87ac0e02d8dc3994333097c268ba2a5dde78308150864e86cb5cb1c279d33aae"}
Jul 21 17:26:48.588911 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:48.588875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:48.589483 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.589034    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:48.589483 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.589109    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs podName:a3f8ea96-7aec-48b2-85ef-6fe96c80ce62 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:20.58908268 +0000 UTC m=+66.147881643 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs") pod "network-metrics-daemon-f6xmv" (UID: "a3f8ea96-7aec-48b2-85ef-6fe96c80ce62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:48.690117 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:48.690091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:48.690268 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.690248    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:48.690338 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.690327    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret podName:1511f43e-7fd8-4f20-86ca-f9eb3c032ac5 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:20.690305665 +0000 UTC m=+66.249104636 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret") pod "global-pull-secret-syncer-l5mfk" (UID: "1511f43e-7fd8-4f20-86ca-f9eb3c032ac5") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:48.791566 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:48.791537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dkl\" (UniqueName: \"kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl\") pod \"network-check-target-8vgqn\" (UID: \"53d47a1a-f442-4cac-abd3-941e47645e03\") " pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:48.791752 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.791699    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:48.791752 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.791723    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:48.791752 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.791736    2579 projected.go:196] Error preparing data for projected volume kube-api-access-w6dkl for pod openshift-network-diagnostics/network-check-target-8vgqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:48.791878 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.791798    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl podName:53d47a1a-f442-4cac-abd3-941e47645e03 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:20.791779845 +0000 UTC m=+66.350578810 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6dkl" (UniqueName: "kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl") pod "network-check-target-8vgqn" (UID: "53d47a1a-f442-4cac-abd3-941e47645e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:48.998542 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:48.998510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:48.998726 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:48.998510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:48.998726 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.998629    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6xmv" podUID="a3f8ea96-7aec-48b2-85ef-6fe96c80ce62"
Jul 21 17:26:48.998726 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.998704    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5mfk" podUID="1511f43e-7fd8-4f20-86ca-f9eb3c032ac5"
Jul 21 17:26:48.998726 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:48.998510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:48.998887 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:48.998781    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vgqn" podUID="53d47a1a-f442-4cac-abd3-941e47645e03"
Jul 21 17:26:50.292820 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.292796    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-31.ec2.internal" event="NodeReady"
Jul 21 17:26:50.293197 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.292908    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 21 17:26:50.348808 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.348780    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ccjk8"]
Jul 21 17:26:50.353964 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.353939    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.355308 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.355283    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gclnx"]
Jul 21 17:26:50.358401 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.358381    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 21 17:26:50.358838 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.358820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gclnx"
Jul 21 17:26:50.362015 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.361451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s47cz\""
Jul 21 17:26:50.362015 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.361708    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 21 17:26:50.363959 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.363937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-brt6b\""
Jul 21 17:26:50.364052 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.364030    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 21 17:26:50.364145 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.364131    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 21 17:26:50.364211 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.364197    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 21 17:26:50.371684 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.371664    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ccjk8"]
Jul 21 17:26:50.383809 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.383788    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gclnx"]
Jul 21 17:26:50.505415 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.505385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7000eef1-3ae1-4925-950d-4929f1c689d1-cert\") pod \"ingress-canary-gclnx\" (UID: \"7000eef1-3ae1-4925-950d-4929f1c689d1\") " pod="openshift-ingress-canary/ingress-canary-gclnx"
Jul 21 17:26:50.505584 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.505427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hl9d\" (UniqueName: \"kubernetes.io/projected/7000eef1-3ae1-4925-950d-4929f1c689d1-kube-api-access-9hl9d\") pod \"ingress-canary-gclnx\" (UID: \"7000eef1-3ae1-4925-950d-4929f1c689d1\") " pod="openshift-ingress-canary/ingress-canary-gclnx"
Jul 21 17:26:50.505584 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.505459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.505584 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.505541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfnq8\" (UniqueName: \"kubernetes.io/projected/20be539f-d29f-4ed6-8c74-1635fdea9bd3-kube-api-access-jfnq8\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.505742 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.505640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20be539f-d29f-4ed6-8c74-1635fdea9bd3-config-volume\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.505742 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.505695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20be539f-d29f-4ed6-8c74-1635fdea9bd3-tmp-dir\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.606743 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.606709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20be539f-d29f-4ed6-8c74-1635fdea9bd3-config-volume\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.606916 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.606760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20be539f-d29f-4ed6-8c74-1635fdea9bd3-tmp-dir\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.606916 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.606784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7000eef1-3ae1-4925-950d-4929f1c689d1-cert\") pod \"ingress-canary-gclnx\" (UID: \"7000eef1-3ae1-4925-950d-4929f1c689d1\") " pod="openshift-ingress-canary/ingress-canary-gclnx"
Jul 21 17:26:50.606916 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.606806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hl9d\" (UniqueName: \"kubernetes.io/projected/7000eef1-3ae1-4925-950d-4929f1c689d1-kube-api-access-9hl9d\") pod \"ingress-canary-gclnx\" (UID: \"7000eef1-3ae1-4925-950d-4929f1c689d1\") " pod="openshift-ingress-canary/ingress-canary-gclnx"
Jul 21 17:26:50.606916 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.606836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.606916 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.606880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnq8\" (UniqueName: \"kubernetes.io/projected/20be539f-d29f-4ed6-8c74-1635fdea9bd3-kube-api-access-jfnq8\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.607172 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:50.607120    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:50.607224 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:50.607180    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls podName:20be539f-d29f-4ed6-8c74-1635fdea9bd3 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:51.107161136 +0000 UTC m=+36.665960112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls") pod "dns-default-ccjk8" (UID: "20be539f-d29f-4ed6-8c74-1635fdea9bd3") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:50.607410 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.607361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20be539f-d29f-4ed6-8c74-1635fdea9bd3-config-volume\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.607542 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.607521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20be539f-d29f-4ed6-8c74-1635fdea9bd3-tmp-dir\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.612139 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.612117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7000eef1-3ae1-4925-950d-4929f1c689d1-cert\") pod \"ingress-canary-gclnx\" (UID: \"7000eef1-3ae1-4925-950d-4929f1c689d1\") " pod="openshift-ingress-canary/ingress-canary-gclnx"
Jul 21 17:26:50.616455 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.616408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnq8\" (UniqueName: \"kubernetes.io/projected/20be539f-d29f-4ed6-8c74-1635fdea9bd3-kube-api-access-jfnq8\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:50.616569 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.616484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hl9d\" (UniqueName: \"kubernetes.io/projected/7000eef1-3ae1-4925-950d-4929f1c689d1-kube-api-access-9hl9d\") pod \"ingress-canary-gclnx\" (UID: \"7000eef1-3ae1-4925-950d-4929f1c689d1\") " pod="openshift-ingress-canary/ingress-canary-gclnx"
Jul 21 17:26:50.676610 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.676581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gclnx"
Jul 21 17:26:50.999006 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.998835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:26:51.001623 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.998835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:26:51.001623 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:50.998877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:26:51.003170 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.002773    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 21 17:26:51.003170 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.003032    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 21 17:26:51.005354 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.004325    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 21 17:26:51.005354 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.004520    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 21 17:26:51.005354 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.004724    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9vqxq\""
Jul 21 17:26:51.005354 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.005077    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfsp8\""
Jul 21 17:26:51.084608 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.084585    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gclnx"]
Jul 21 17:26:51.088232 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:51.088204    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7000eef1_3ae1_4925_950d_4929f1c689d1.slice/crio-e6b71385a2afa926b1344d4aabed73c4a5f3b1853c4335d5b84084856548ec47 WatchSource:0}: Error finding container e6b71385a2afa926b1344d4aabed73c4a5f3b1853c4335d5b84084856548ec47: Status 404 returned error can't find the container with id e6b71385a2afa926b1344d4aabed73c4a5f3b1853c4335d5b84084856548ec47
Jul 21 17:26:51.110324 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.110302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:51.110513 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:51.110487    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:51.110589 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:51.110576    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls podName:20be539f-d29f-4ed6-8c74-1635fdea9bd3 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:52.110553668 +0000 UTC m=+37.669352685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls") pod "dns-default-ccjk8" (UID: "20be539f-d29f-4ed6-8c74-1635fdea9bd3") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:51.149940 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.149900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gclnx" event={"ID":"7000eef1-3ae1-4925-950d-4929f1c689d1","Type":"ContainerStarted","Data":"e6b71385a2afa926b1344d4aabed73c4a5f3b1853c4335d5b84084856548ec47"}
Jul 21 17:26:51.152860 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.152838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-67qsr" event={"ID":"5dad033f-a636-412a-81fc-3e7b683e14e8","Type":"ContainerStarted","Data":"11e9abd4f2241c5b72159f3de705f606738f1fe39021877938d9bcab0f3203bd"}
Jul 21 17:26:51.734742 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.734711    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp"]
Jul 21 17:26:51.738025 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.738006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp"
Jul 21 17:26:51.738145 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.738030    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"]
Jul 21 17:26:51.741221 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.741154    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 21 17:26:51.741335 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.741224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:26:51.741468 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.741446    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:26:51.742702 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.742682    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-sbfxc\""
Jul 21 17:26:51.743831 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.743808    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 21 17:26:51.744356 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.744334    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:26:51.744873 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.744852    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ffp47\""
Jul 21 17:26:51.744975 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.744857    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 21 17:26:51.747416 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.747364    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp"]
Jul 21 17:26:51.753134 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.752448    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"]
Jul 21 17:26:51.756084 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.755947    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"]
Jul 21 17:26:51.756210 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.756091    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:51.759378 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.759321    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:26:51.759378 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.759368    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 21 17:26:51.759561 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.759388    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 21 17:26:51.759561 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.759323    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pnf84\""
Jul 21 17:26:51.760122 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.759985    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 21 17:26:51.766515 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.766495    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"]
Jul 21 17:26:51.814233 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.814209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:26:51.814354 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.814239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mjq\" (UniqueName: \"kubernetes.io/projected/f123a999-8a3e-4833-a0b9-3688623cceb3-kube-api-access-d9mjq\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:26:51.814354 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.814342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdp7\" (UniqueName: \"kubernetes.io/projected/42597b38-e5e4-42f6-8599-3d2d5645dbf5-kube-api-access-8hdp7\") pod \"volume-data-source-validator-78cf4c6f6-jbwcp\" (UID: \"42597b38-e5e4-42f6-8599-3d2d5645dbf5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp"
Jul 21 17:26:51.842014 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.841990    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-9f678b54d-lmmtf"]
Jul 21 17:26:51.845233 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.845216    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d55d46bc-pj2hs"]
Jul 21 17:26:51.845354 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.845335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:51.848353 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.848327    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-x2sft"]
Jul 21 17:26:51.848462 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.848446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:51.851214 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.851191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:51.855910 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.855891    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 21 17:26:51.856207 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.856189    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 21 17:26:51.856207 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.856198    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 21 17:26:51.856470 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.856452    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c8xpg\""
Jul 21 17:26:51.856961 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.856946    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 21 17:26:51.858500 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.858481    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 21 17:26:51.860413 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.860392    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 21 17:26:51.860516 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.860422    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 21 17:26:51.860696 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.860678    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 21 17:26:51.860800 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.860753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 21 17:26:51.860800 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.860777    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 21 17:26:51.860911 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.860874    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 21 17:26:51.865906 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.865805    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 21 17:26:51.866411 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.866395    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bwwcr\""
Jul 21 17:26:51.867145 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.867117    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d6tcj\""
Jul 21 17:26:51.867247 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.867162    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 21 17:26:51.879786 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.879764    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 21 17:26:51.882243 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.882223    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-x2sft"]
Jul 21 17:26:51.899379 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.899348    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 21 17:26:51.899473 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.899407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9f678b54d-lmmtf"]
Jul 21 17:26:51.911799 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.911774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d55d46bc-pj2hs"]
Jul 21 17:26:51.915181 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdp7\" (UniqueName: \"kubernetes.io/projected/42597b38-e5e4-42f6-8599-3d2d5645dbf5-kube-api-access-8hdp7\") pod \"volume-data-source-validator-78cf4c6f6-jbwcp\" (UID: \"42597b38-e5e4-42f6-8599-3d2d5645dbf5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp"
Jul 21 17:26:51.915537 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86fg\" (UniqueName: \"kubernetes.io/projected/3f5d4bf0-deac-486f-a3dd-a5ec092c342d-kube-api-access-p86fg\") pod \"kube-storage-version-migrator-operator-7658d9b749-prmlj\" (UID: \"3f5d4bf0-deac-486f-a3dd-a5ec092c342d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:51.915537 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-image-registry-private-configuration\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:51.915537 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f5d4bf0-deac-486f-a3dd-a5ec092c342d-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-prmlj\" (UID: \"3f5d4bf0-deac-486f-a3dd-a5ec092c342d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:51.915537 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r4zg\" (UniqueName: \"kubernetes.io/projected/e56663cb-5de7-4266-b7ca-1aa0bf94928a-kube-api-access-4r4zg\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:51.915537 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8677903c-34d1-41e8-b0c7-be1e3188b4ce-serving-cert\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:51.915537 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:51.915537 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8677903c-34d1-41e8-b0c7-be1e3188b4ce-snapshots\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:51.915537 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mjq\" (UniqueName: \"kubernetes.io/projected/f123a999-8a3e-4833-a0b9-3688623cceb3-kube-api-access-d9mjq\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:26:51.915537 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-installation-pull-secrets\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:51.915975 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:26:51.915975 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8677903c-34d1-41e8-b0c7-be1e3188b4ce-tmp\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:51.915975 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-stats-auth\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:51.915975 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-default-certificate\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:51.915975 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:51.915840    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 21 17:26:51.915975 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8677903c-34d1-41e8-b0c7-be1e3188b4ce-service-ca-bundle\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:51.915975 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:51.915885    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls podName:f123a999-8a3e-4833-a0b9-3688623cceb3 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:52.415871264 +0000 UTC m=+37.974670225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-z58mb" (UID: "f123a999-8a3e-4833-a0b9-3688623cceb3") : secret "samples-operator-tls" not found
Jul 21 17:26:51.915975 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8677903c-34d1-41e8-b0c7-be1e3188b4ce-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:51.915975 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-certificates\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:51.916295 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.915998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:51.916295 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.916027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb9tf\" (UniqueName: \"kubernetes.io/projected/8677903c-34d1-41e8-b0c7-be1e3188b4ce-kube-api-access-lb9tf\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:51.916295 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.916050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgr69\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-kube-api-access-vgr69\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:51.916295 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.916073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-bound-sa-token\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:51.916295 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.916097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f5d4bf0-deac-486f-a3dd-a5ec092c342d-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-prmlj\" (UID: \"3f5d4bf0-deac-486f-a3dd-a5ec092c342d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:51.916295 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.916120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:51.916295 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.916163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-trusted-ca\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:51.916295 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.916221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f981f7f-a854-4b87-9c61-8595094ba7cb-ca-trust-extracted\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:51.942300 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.942278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mjq\" (UniqueName: \"kubernetes.io/projected/f123a999-8a3e-4833-a0b9-3688623cceb3-kube-api-access-d9mjq\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:26:51.950003 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:51.949981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdp7\" (UniqueName: \"kubernetes.io/projected/42597b38-e5e4-42f6-8599-3d2d5645dbf5-kube-api-access-8hdp7\") pod \"volume-data-source-validator-78cf4c6f6-jbwcp\" (UID: \"42597b38-e5e4-42f6-8599-3d2d5645dbf5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp"
Jul 21 17:26:52.017295 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8677903c-34d1-41e8-b0c7-be1e3188b4ce-tmp\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.017415 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-stats-auth\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:52.017415 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-default-certificate\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:52.017528 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8677903c-34d1-41e8-b0c7-be1e3188b4ce-service-ca-bundle\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.017528 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8677903c-34d1-41e8-b0c7-be1e3188b4ce-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.017629 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-certificates\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.017629 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.017629 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb9tf\" (UniqueName: \"kubernetes.io/projected/8677903c-34d1-41e8-b0c7-be1e3188b4ce-kube-api-access-lb9tf\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.017800 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgr69\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-kube-api-access-vgr69\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.017800 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-bound-sa-token\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.017800 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017688    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8677903c-34d1-41e8-b0c7-be1e3188b4ce-tmp\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.017800 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f5d4bf0-deac-486f-a3dd-a5ec092c342d-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-prmlj\" (UID: \"3f5d4bf0-deac-486f-a3dd-a5ec092c342d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:52.017800 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:52.017800 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-trusted-ca\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.017800 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f981f7f-a854-4b87-9c61-8595094ba7cb-ca-trust-extracted\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.018073 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.017809    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 17:26:52.018073 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.017824    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d55d46bc-pj2hs: secret "image-registry-tls" not found
Jul 21 17:26:52.018073 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86fg\" (UniqueName: \"kubernetes.io/projected/3f5d4bf0-deac-486f-a3dd-a5ec092c342d-kube-api-access-p86fg\") pod \"kube-storage-version-migrator-operator-7658d9b749-prmlj\" (UID: \"3f5d4bf0-deac-486f-a3dd-a5ec092c342d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:52.018073 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.017874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-image-registry-private-configuration\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.018228 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.018211    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls podName:3f981f7f-a854-4b87-9c61-8595094ba7cb nodeName:}" failed. No retries permitted until 2026-07-21 17:26:52.518167274 +0000 UTC m=+38.076966256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls") pod "image-registry-6d55d46bc-pj2hs" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb") : secret "image-registry-tls" not found
Jul 21 17:26:52.019933 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.019912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f5d4bf0-deac-486f-a3dd-a5ec092c342d-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-prmlj\" (UID: \"3f5d4bf0-deac-486f-a3dd-a5ec092c342d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:52.020032 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.019962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r4zg\" (UniqueName: \"kubernetes.io/projected/e56663cb-5de7-4266-b7ca-1aa0bf94928a-kube-api-access-4r4zg\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:52.020032 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.020002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8677903c-34d1-41e8-b0c7-be1e3188b4ce-serving-cert\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.020137 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.020054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:52.020137 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.020088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8677903c-34d1-41e8-b0c7-be1e3188b4ce-snapshots\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.020235 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.020145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-installation-pull-secrets\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.023370 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.021357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-certificates\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.023370 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.021475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-stats-auth\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:52.023370 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.021867    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:26:52.023370 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.021925    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:52.521907465 +0000 UTC m=+38.080706440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : secret "router-metrics-certs-default" not found
Jul 21 17:26:52.023370 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.022032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8677903c-34d1-41e8-b0c7-be1e3188b4ce-snapshots\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.023370 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.022159    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:52.522141431 +0000 UTC m=+38.080940417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : configmap references non-existent config key: service-ca.crt
Jul 21 17:26:52.023370 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.022292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f5d4bf0-deac-486f-a3dd-a5ec092c342d-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-prmlj\" (UID: \"3f5d4bf0-deac-486f-a3dd-a5ec092c342d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:52.023370 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.022449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f981f7f-a854-4b87-9c61-8595094ba7cb-ca-trust-extracted\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.023370 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.022947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-installation-pull-secrets\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.023370 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.023040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8677903c-34d1-41e8-b0c7-be1e3188b4ce-service-ca-bundle\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.024868 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.024836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-trusted-ca\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.027993 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.025524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8677903c-34d1-41e8-b0c7-be1e3188b4ce-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.028123 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.026857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f5d4bf0-deac-486f-a3dd-a5ec092c342d-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-prmlj\" (UID: \"3f5d4bf0-deac-486f-a3dd-a5ec092c342d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:52.028123 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.026922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-default-certificate\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:52.028281 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.027722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-image-registry-private-configuration\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.028281 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.027926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8677903c-34d1-41e8-b0c7-be1e3188b4ce-serving-cert\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.037613 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.037592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb9tf\" (UniqueName: \"kubernetes.io/projected/8677903c-34d1-41e8-b0c7-be1e3188b4ce-kube-api-access-lb9tf\") pod \"insights-operator-555d9d7885-x2sft\" (UID: \"8677903c-34d1-41e8-b0c7-be1e3188b4ce\") " pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.038495 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.038471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgr69\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-kube-api-access-vgr69\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.039220 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.039188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-bound-sa-token\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.039346 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.039330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86fg\" (UniqueName: \"kubernetes.io/projected/3f5d4bf0-deac-486f-a3dd-a5ec092c342d-kube-api-access-p86fg\") pod \"kube-storage-version-migrator-operator-7658d9b749-prmlj\" (UID: \"3f5d4bf0-deac-486f-a3dd-a5ec092c342d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:52.040207 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.040190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r4zg\" (UniqueName: \"kubernetes.io/projected/e56663cb-5de7-4266-b7ca-1aa0bf94928a-kube-api-access-4r4zg\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:52.053195 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.053176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp"
Jul 21 17:26:52.070912 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.070892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"
Jul 21 17:26:52.122808 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.122039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:52.122808 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.122309    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:52.122808 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.122371    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls podName:20be539f-d29f-4ed6-8c74-1635fdea9bd3 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:54.122351033 +0000 UTC m=+39.681149997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls") pod "dns-default-ccjk8" (UID: "20be539f-d29f-4ed6-8c74-1635fdea9bd3") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:52.160112 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.159946    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dad033f-a636-412a-81fc-3e7b683e14e8" containerID="11e9abd4f2241c5b72159f3de705f606738f1fe39021877938d9bcab0f3203bd" exitCode=0
Jul 21 17:26:52.160112 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.159995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-67qsr" event={"ID":"5dad033f-a636-412a-81fc-3e7b683e14e8","Type":"ContainerDied","Data":"11e9abd4f2241c5b72159f3de705f606738f1fe39021877938d9bcab0f3203bd"}
Jul 21 17:26:52.168939 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.168761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-x2sft"
Jul 21 17:26:52.220395 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.220323    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp"]
Jul 21 17:26:52.226856 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:52.226820    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42597b38_e5e4_42f6_8599_3d2d5645dbf5.slice/crio-bc17b338a86d6abe804a67376a3041c1b1dcc3888b0cdf903b46072a6d3532c9 WatchSource:0}: Error finding container bc17b338a86d6abe804a67376a3041c1b1dcc3888b0cdf903b46072a6d3532c9: Status 404 returned error can't find the container with id bc17b338a86d6abe804a67376a3041c1b1dcc3888b0cdf903b46072a6d3532c9
Jul 21 17:26:52.236988 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.236794    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj"]
Jul 21 17:26:52.240414 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:52.240371    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5d4bf0_deac_486f_a3dd_a5ec092c342d.slice/crio-bafdd75d7d873860e5f1e642c437bb545e80a9a06ca9b98c9a1464b0e7b752e7 WatchSource:0}: Error finding container bafdd75d7d873860e5f1e642c437bb545e80a9a06ca9b98c9a1464b0e7b752e7: Status 404 returned error can't find the container with id bafdd75d7d873860e5f1e642c437bb545e80a9a06ca9b98c9a1464b0e7b752e7
Jul 21 17:26:52.320908 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.320885    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-x2sft"]
Jul 21 17:26:52.368666 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:26:52.368619    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8677903c_34d1_41e8_b0c7_be1e3188b4ce.slice/crio-4d1b9fd5aa5ee6984a0d6ac3b4d7ab4e259a1d144d090c6df9ed8bbe6f1c9018 WatchSource:0}: Error finding container 4d1b9fd5aa5ee6984a0d6ac3b4d7ab4e259a1d144d090c6df9ed8bbe6f1c9018: Status 404 returned error can't find the container with id 4d1b9fd5aa5ee6984a0d6ac3b4d7ab4e259a1d144d090c6df9ed8bbe6f1c9018
Jul 21 17:26:52.425259 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.425200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:26:52.425381 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.425361    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 21 17:26:52.425432 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.425423    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls podName:f123a999-8a3e-4833-a0b9-3688623cceb3 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:53.425408435 +0000 UTC m=+38.984207398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-z58mb" (UID: "f123a999-8a3e-4833-a0b9-3688623cceb3") : secret "samples-operator-tls" not found
Jul 21 17:26:52.526050 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.526018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:52.526217 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.526146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:52.526217 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.526183    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:53.52616511 +0000 UTC m=+39.084964072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : configmap references non-existent config key: service-ca.crt
Jul 21 17:26:52.526340 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:52.526215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:52.526340 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.526283    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 17:26:52.526340 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.526298    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d55d46bc-pj2hs: secret "image-registry-tls" not found
Jul 21 17:26:52.526340 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.526298    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:26:52.526340 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.526332    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:53.526324533 +0000 UTC m=+39.085123494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : secret "router-metrics-certs-default" not found
Jul 21 17:26:52.526546 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:52.526346    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls podName:3f981f7f-a854-4b87-9c61-8595094ba7cb nodeName:}" failed. No retries permitted until 2026-07-21 17:26:53.526338669 +0000 UTC m=+39.085137632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls") pod "image-registry-6d55d46bc-pj2hs" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb") : secret "image-registry-tls" not found
Jul 21 17:26:53.164072 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:53.164035    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-x2sft" event={"ID":"8677903c-34d1-41e8-b0c7-be1e3188b4ce","Type":"ContainerStarted","Data":"4d1b9fd5aa5ee6984a0d6ac3b4d7ab4e259a1d144d090c6df9ed8bbe6f1c9018"}
Jul 21 17:26:53.166193 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:53.166164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp" event={"ID":"42597b38-e5e4-42f6-8599-3d2d5645dbf5","Type":"ContainerStarted","Data":"bc17b338a86d6abe804a67376a3041c1b1dcc3888b0cdf903b46072a6d3532c9"}
Jul 21 17:26:53.171591 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:53.170949    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dad033f-a636-412a-81fc-3e7b683e14e8" containerID="a0a935f38e1e2bd4c9d3b129773515a007f8ff224ded50fe3d77f5929c396132" exitCode=0
Jul 21 17:26:53.171591 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:53.171024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-67qsr" event={"ID":"5dad033f-a636-412a-81fc-3e7b683e14e8","Type":"ContainerDied","Data":"a0a935f38e1e2bd4c9d3b129773515a007f8ff224ded50fe3d77f5929c396132"}
Jul 21 17:26:53.173635 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:53.173567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj" event={"ID":"3f5d4bf0-deac-486f-a3dd-a5ec092c342d","Type":"ContainerStarted","Data":"bafdd75d7d873860e5f1e642c437bb545e80a9a06ca9b98c9a1464b0e7b752e7"}
Jul 21 17:26:53.436051 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:53.435928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:26:53.436207 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:53.436096    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 21 17:26:53.436207 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:53.436181    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls podName:f123a999-8a3e-4833-a0b9-3688623cceb3 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:55.436159828 +0000 UTC m=+40.994958811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-z58mb" (UID: "f123a999-8a3e-4833-a0b9-3688623cceb3") : secret "samples-operator-tls" not found
Jul 21 17:26:53.536838 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:53.536800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:53.537008 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:53.536856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:53.537008 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:53.536911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:53.537122 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:53.537019    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:26:53.537122 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:53.537091    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:55.537064394 +0000 UTC m=+41.095863374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : secret "router-metrics-certs-default" not found
Jul 21 17:26:53.537122 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:53.537111    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:55.537101835 +0000 UTC m=+41.095900801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : configmap references non-existent config key: service-ca.crt
Jul 21 17:26:53.537291 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:53.537128    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 17:26:53.537291 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:53.537145    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d55d46bc-pj2hs: secret "image-registry-tls" not found
Jul 21 17:26:53.537291 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:53.537188    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls podName:3f981f7f-a854-4b87-9c61-8595094ba7cb nodeName:}" failed. No retries permitted until 2026-07-21 17:26:55.53717272 +0000 UTC m=+41.095971684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls") pod "image-registry-6d55d46bc-pj2hs" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb") : secret "image-registry-tls" not found
Jul 21 17:26:54.142685 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:54.142641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:54.142856 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:54.142788    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:54.142902 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:54.142879    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls podName:20be539f-d29f-4ed6-8c74-1635fdea9bd3 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:58.142856149 +0000 UTC m=+43.701655119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls") pod "dns-default-ccjk8" (UID: "20be539f-d29f-4ed6-8c74-1635fdea9bd3") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:55.453122 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:55.453071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:26:55.453796 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:55.453245    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 21 17:26:55.453796 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:55.453323    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls podName:f123a999-8a3e-4833-a0b9-3688623cceb3 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:59.453301934 +0000 UTC m=+45.012100901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-z58mb" (UID: "f123a999-8a3e-4833-a0b9-3688623cceb3") : secret "samples-operator-tls" not found
Jul 21 17:26:55.553713 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:55.553684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:55.553824 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:55.553720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:55.553880 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:55.553826    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:26:55.553880 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:55.553830    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 17:26:55.553880 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:55.553846    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d55d46bc-pj2hs: secret "image-registry-tls" not found
Jul 21 17:26:55.553880 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:55.553879    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:59.553863852 +0000 UTC m=+45.112662814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : secret "router-metrics-certs-default" not found
Jul 21 17:26:55.554069 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:55.553879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:55.554069 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:55.553897    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls podName:3f981f7f-a854-4b87-9c61-8595094ba7cb nodeName:}" failed. No retries permitted until 2026-07-21 17:26:59.5538882 +0000 UTC m=+45.112687167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls") pod "image-registry-6d55d46bc-pj2hs" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb") : secret "image-registry-tls" not found
Jul 21 17:26:55.554069 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:55.553996    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:59.553978418 +0000 UTC m=+45.112777391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : configmap references non-existent config key: service-ca.crt
Jul 21 17:26:56.183856 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:56.183819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gclnx" event={"ID":"7000eef1-3ae1-4925-950d-4929f1c689d1","Type":"ContainerStarted","Data":"4f26b703956ac08ad85a19e0ba44632bde886fd03e04045fe22d86b287e21927"}
Jul 21 17:26:56.185338 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:56.185315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-x2sft" event={"ID":"8677903c-34d1-41e8-b0c7-be1e3188b4ce","Type":"ContainerStarted","Data":"732f95a4adb0b2640122a8f090ffef07a2510332d533d4b152241d103ffa473f"}
Jul 21 17:26:56.186925 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:56.186901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp" event={"ID":"42597b38-e5e4-42f6-8599-3d2d5645dbf5","Type":"ContainerStarted","Data":"adb362caed95e71bb68164a5318f9a448560ce1fc88f6e3353efd2b6c524158f"}
Jul 21 17:26:56.190404 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:56.190367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-67qsr" event={"ID":"5dad033f-a636-412a-81fc-3e7b683e14e8","Type":"ContainerStarted","Data":"adc2e57be9b0099962cc13ee51dd984892905ece0b2edcecd7fe96ee33c41ba9"}
Jul 21 17:26:56.191884 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:56.191861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj" event={"ID":"3f5d4bf0-deac-486f-a3dd-a5ec092c342d","Type":"ContainerStarted","Data":"815fcb52488ea7be7780f38bbd41a2b72a055b0f459ff7b5dd92bb612cf56ffa"}
Jul 21 17:26:56.200911 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:56.200873    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gclnx" podStartSLOduration=1.597967411 podStartE2EDuration="6.200860334s" totalImagesPullingTime="4.602892923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:56.199604164 +0000 UTC m=+41.758403150" watchObservedRunningTime="2026-07-21 17:26:56.200860334 +0000 UTC m=+41.759659343"
Jul 21 17:26:56.223497 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:56.223449    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-67qsr" podStartSLOduration=11.137409416 podStartE2EDuration="41.223436729s" totalImagesPullingTime="30.086027313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:56.22299634 +0000 UTC m=+41.781795341" watchObservedRunningTime="2026-07-21 17:26:56.223436729 +0000 UTC m=+41.782235712"
Jul 21 17:26:56.241053 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:56.241016    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-x2sft" podStartSLOduration=1.915397087 podStartE2EDuration="5.241005059s" totalImagesPullingTime="3.325607972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:56.240523989 +0000 UTC m=+41.799322974" watchObservedRunningTime="2026-07-21 17:26:56.241005059 +0000 UTC m=+41.799804046"
Jul 21 17:26:56.257349 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:56.256835    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj" podStartSLOduration=1.806629122 podStartE2EDuration="5.256820602s" totalImagesPullingTime="3.45019148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:56.256544344 +0000 UTC m=+41.815343329" watchObservedRunningTime="2026-07-21 17:26:56.256820602 +0000 UTC m=+41.815619596"
Jul 21 17:26:56.275857 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:56.275816    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jbwcp" podStartSLOduration=1.811679337 podStartE2EDuration="5.275801988s" totalImagesPullingTime="3.464122651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:56.275195939 +0000 UTC m=+41.833994945" watchObservedRunningTime="2026-07-21 17:26:56.275801988 +0000 UTC m=+41.834600974"
Jul 21 17:26:57.108157 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.108123    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v92lz"]
Jul 21 17:26:57.141226 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.141201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.149559 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.149538    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 21 17:26:57.149700 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.149625    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 21 17:26:57.149780 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.149766    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2qswg\""
Jul 21 17:26:57.174258 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.174235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v92lz"]
Jul 21 17:26:57.268523 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.268494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10f013b0-411e-4cd7-b170-2dd308190cea-data-volume\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.268673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.268603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.268673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.268647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.268751 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.268690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnkw\" (UniqueName: \"kubernetes.io/projected/10f013b0-411e-4cd7-b170-2dd308190cea-kube-api-access-wlnkw\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.268788 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.268755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10f013b0-411e-4cd7-b170-2dd308190cea-crio-socket\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.369308 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.369234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10f013b0-411e-4cd7-b170-2dd308190cea-data-volume\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.369615 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.369591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10f013b0-411e-4cd7-b170-2dd308190cea-data-volume\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.369723 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.369613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.369723 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.369709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.369810 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:57.369735    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:26:57.369810 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:57.369747    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:26:57.369810 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.369765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnkw\" (UniqueName: \"kubernetes.io/projected/10f013b0-411e-4cd7-b170-2dd308190cea-kube-api-access-wlnkw\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.369810 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:57.369797    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls podName:10f013b0-411e-4cd7-b170-2dd308190cea nodeName:}" failed. No retries permitted until 2026-07-21 17:26:57.86977536 +0000 UTC m=+43.428574332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v92lz" (UID: "10f013b0-411e-4cd7-b170-2dd308190cea") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:26:57.370020 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:57.369865    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm podName:10f013b0-411e-4cd7-b170-2dd308190cea nodeName:}" failed. No retries permitted until 2026-07-21 17:26:57.869847953 +0000 UTC m=+43.428646929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v92lz" (UID: "10f013b0-411e-4cd7-b170-2dd308190cea") : configmap "kube-rbac-proxy" not found
Jul 21 17:26:57.370020 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.369930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10f013b0-411e-4cd7-b170-2dd308190cea-crio-socket\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.370020 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.369969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10f013b0-411e-4cd7-b170-2dd308190cea-crio-socket\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.380385 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.380363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnkw\" (UniqueName: \"kubernetes.io/projected/10f013b0-411e-4cd7-b170-2dd308190cea-kube-api-access-wlnkw\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.873614 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.873577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.873784 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:57.873631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:57.873784 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:57.873769    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:26:57.873866 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:57.873828    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm podName:10f013b0-411e-4cd7-b170-2dd308190cea nodeName:}" failed. No retries permitted until 2026-07-21 17:26:58.873813202 +0000 UTC m=+44.432612165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v92lz" (UID: "10f013b0-411e-4cd7-b170-2dd308190cea") : configmap "kube-rbac-proxy" not found
Jul 21 17:26:57.873866 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:57.873769    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:26:57.873956 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:57.873899    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls podName:10f013b0-411e-4cd7-b170-2dd308190cea nodeName:}" failed. No retries permitted until 2026-07-21 17:26:58.873886642 +0000 UTC m=+44.432685608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v92lz" (UID: "10f013b0-411e-4cd7-b170-2dd308190cea") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:26:58.175670 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:58.175567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:26:58.176052 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:58.175731    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:58.176052 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:58.175797    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls podName:20be539f-d29f-4ed6-8c74-1635fdea9bd3 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:06.175780402 +0000 UTC m=+51.734579364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls") pod "dns-default-ccjk8" (UID: "20be539f-d29f-4ed6-8c74-1635fdea9bd3") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:58.881397 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:58.881362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:58.881567 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:58.881412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:26:58.881567 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:58.881481    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:26:58.881567 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:58.881534    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm podName:10f013b0-411e-4cd7-b170-2dd308190cea nodeName:}" failed. No retries permitted until 2026-07-21 17:27:00.881520553 +0000 UTC m=+46.440319516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v92lz" (UID: "10f013b0-411e-4cd7-b170-2dd308190cea") : configmap "kube-rbac-proxy" not found
Jul 21 17:26:58.881701 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:58.881567    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:26:58.881701 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:58.881616    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls podName:10f013b0-411e-4cd7-b170-2dd308190cea nodeName:}" failed. No retries permitted until 2026-07-21 17:27:00.881604379 +0000 UTC m=+46.440403340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v92lz" (UID: "10f013b0-411e-4cd7-b170-2dd308190cea") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:26:59.488620 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:59.488583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:26:59.488998 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:59.488745    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 21 17:26:59.488998 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:59.488808    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls podName:f123a999-8a3e-4833-a0b9-3688623cceb3 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:07.488792469 +0000 UTC m=+53.047591431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-z58mb" (UID: "f123a999-8a3e-4833-a0b9-3688623cceb3") : secret "samples-operator-tls" not found
Jul 21 17:26:59.588993 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:59.588967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:26:59.589120 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:59.588999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:59.589120 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:26:59.589033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:26:59.589120 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:59.589112    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 17:26:59.589218 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:59.589116    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:26:59.589218 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:59.589127    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d55d46bc-pj2hs: secret "image-registry-tls" not found
Jul 21 17:26:59.589218 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:59.589162    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:07.589148016 +0000 UTC m=+53.147946980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : configmap references non-existent config key: service-ca.crt
Jul 21 17:26:59.589218 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:59.589177    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:07.58917122 +0000 UTC m=+53.147970182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : secret "router-metrics-certs-default" not found
Jul 21 17:26:59.589218 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:26:59.589189    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls podName:3f981f7f-a854-4b87-9c61-8595094ba7cb nodeName:}" failed. No retries permitted until 2026-07-21 17:27:07.589184208 +0000 UTC m=+53.147983169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls") pod "image-registry-6d55d46bc-pj2hs" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb") : secret "image-registry-tls" not found
Jul 21 17:27:00.117804 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:00.117777    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mz828_3b03ab87-1a4b-4b9d-b530-e3c8d082464d/dns-node-resolver/0.log"
Jul 21 17:27:00.899574 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:00.899537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:27:00.900010 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:00.899595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:27:00.900010 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:00.899765    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:00.900010 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:00.899842    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls podName:10f013b0-411e-4cd7-b170-2dd308190cea nodeName:}" failed. No retries permitted until 2026-07-21 17:27:04.899820887 +0000 UTC m=+50.458619862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v92lz" (UID: "10f013b0-411e-4cd7-b170-2dd308190cea") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:00.900205 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:00.900187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10f013b0-411e-4cd7-b170-2dd308190cea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:27:01.114059 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:01.114030    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sckv6_59fc317b-41ba-4e6e-8b7c-5ae9112a89cf/node-ca/0.log"
Jul 21 17:27:01.914611 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:01.914583    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gclnx_7000eef1-3ae1-4925-950d-4929f1c689d1/serve-healthcheck-canary/0.log"
Jul 21 17:27:02.517545 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:02.517513    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-prmlj_3f5d4bf0-deac-486f-a3dd-a5ec092c342d/kube-storage-version-migrator-operator/0.log"
Jul 21 17:27:04.934214 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:04.934184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:27:04.934701 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:04.934347    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:04.934701 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:04.934421    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls podName:10f013b0-411e-4cd7-b170-2dd308190cea nodeName:}" failed. No retries permitted until 2026-07-21 17:27:12.93440251 +0000 UTC m=+58.493201474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v92lz" (UID: "10f013b0-411e-4cd7-b170-2dd308190cea") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:06.247041 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:06.247009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:27:06.247563 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:06.247548    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:27:06.247614 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:06.247608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls podName:20be539f-d29f-4ed6-8c74-1635fdea9bd3 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:22.247590977 +0000 UTC m=+67.806389951 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls") pod "dns-default-ccjk8" (UID: "20be539f-d29f-4ed6-8c74-1635fdea9bd3") : secret "dns-default-metrics-tls" not found
Jul 21 17:27:07.556833 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:07.556803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:27:07.559035 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:07.559016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f123a999-8a3e-4833-a0b9-3688623cceb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-z58mb\" (UID: \"f123a999-8a3e-4833-a0b9-3688623cceb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:27:07.657423 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:07.657389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:27:07.657423 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:07.657427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:07.657594 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:07.657474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:07.657594 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:07.657551    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 17:27:07.657594 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:07.657567    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d55d46bc-pj2hs: secret "image-registry-tls" not found
Jul 21 17:27:07.657707 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:07.657606    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:23.657588022 +0000 UTC m=+69.216387011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : configmap references non-existent config key: service-ca.crt
Jul 21 17:27:07.657707 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:07.657613    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:27:07.657707 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:07.657622    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls podName:3f981f7f-a854-4b87-9c61-8595094ba7cb nodeName:}" failed. No retries permitted until 2026-07-21 17:27:23.657615583 +0000 UTC m=+69.216414545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls") pod "image-registry-6d55d46bc-pj2hs" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb") : secret "image-registry-tls" not found
Jul 21 17:27:07.657806 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:27:07.657713    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs podName:e56663cb-5de7-4266-b7ca-1aa0bf94928a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:23.657688096 +0000 UTC m=+69.216487068 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs") pod "router-default-9f678b54d-lmmtf" (UID: "e56663cb-5de7-4266-b7ca-1aa0bf94928a") : secret "router-metrics-certs-default" not found
Jul 21 17:27:07.659366 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:07.659347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"
Jul 21 17:27:07.797474 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:07.797446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb"]
Jul 21 17:27:08.225152 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:08.225122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb" event={"ID":"f123a999-8a3e-4833-a0b9-3688623cceb3","Type":"ContainerStarted","Data":"3c02d5d2d80440ce0946ae36a097761c5559bf7e7fa89c118ab0086455586600"}
Jul 21 17:27:10.232382 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:10.232351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb" event={"ID":"f123a999-8a3e-4833-a0b9-3688623cceb3","Type":"ContainerStarted","Data":"a4874c49d84c99860c34f9c5f91049cf8995f60163ff6ee4e1743749cba0c446"}
Jul 21 17:27:10.232760 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:10.232399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb" event={"ID":"f123a999-8a3e-4833-a0b9-3688623cceb3","Type":"ContainerStarted","Data":"20abd9c6f31bff2eda6a13841368896b9b100f66dcb85811ea4f2fd44208c321"}
Jul 21 17:27:10.255936 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:10.255896    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-z58mb" podStartSLOduration=17.277232865 podStartE2EDuration="19.255881169s" totalImagesPullingTime="1.978648304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:10.255839722 +0000 UTC m=+55.814638705" watchObservedRunningTime="2026-07-21 17:27:10.255881169 +0000 UTC m=+55.814680152"
Jul 21 17:27:13.000273 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:13.000241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:27:13.002698 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:13.002673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10f013b0-411e-4cd7-b170-2dd308190cea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v92lz\" (UID: \"10f013b0-411e-4cd7-b170-2dd308190cea\") " pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:27:13.049569 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:13.049533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v92lz"
Jul 21 17:27:13.173217 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:13.173185    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v92lz"]
Jul 21 17:27:13.176011 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:13.175989    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f013b0_411e_4cd7_b170_2dd308190cea.slice/crio-8fb8f1852be795e42b7ecc07f9bfa32a34074f8a6e0f44a7c6bfa9ebda5c4d33 WatchSource:0}: Error finding container 8fb8f1852be795e42b7ecc07f9bfa32a34074f8a6e0f44a7c6bfa9ebda5c4d33: Status 404 returned error can't find the container with id 8fb8f1852be795e42b7ecc07f9bfa32a34074f8a6e0f44a7c6bfa9ebda5c4d33
Jul 21 17:27:13.242206 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:13.242176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v92lz" event={"ID":"10f013b0-411e-4cd7-b170-2dd308190cea","Type":"ContainerStarted","Data":"8fb8f1852be795e42b7ecc07f9bfa32a34074f8a6e0f44a7c6bfa9ebda5c4d33"}
Jul 21 17:27:16.154245 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:16.154213    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr248"
Jul 21 17:27:18.260583 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:18.260496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v92lz" event={"ID":"10f013b0-411e-4cd7-b170-2dd308190cea","Type":"ContainerStarted","Data":"c749e11e4273ecd37ccc94cab02eada08575d174ded965a3e2c7dbf50e33dd98"}
Jul 21 17:27:20.668282 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.668247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:27:20.671608 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.671590    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 21 17:27:20.681698 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.681677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8ea96-7aec-48b2-85ef-6fe96c80ce62-metrics-certs\") pod \"network-metrics-daemon-f6xmv\" (UID: \"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62\") " pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:27:20.726898 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.726871    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfsp8\""
Jul 21 17:27:20.733584 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.733559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6xmv"
Jul 21 17:27:20.768993 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.768959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:27:20.772825 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.772807    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 21 17:27:20.782159 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.782135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1511f43e-7fd8-4f20-86ca-f9eb3c032ac5-original-pull-secret\") pod \"global-pull-secret-syncer-l5mfk\" (UID: \"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5\") " pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:27:20.869797 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.869762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dkl\" (UniqueName: \"kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl\") pod \"network-check-target-8vgqn\" (UID: \"53d47a1a-f442-4cac-abd3-941e47645e03\") " pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:27:20.873016 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.872992    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 21 17:27:20.883291 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.883270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 21 17:27:20.893635 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:20.893612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dkl\" (UniqueName: \"kubernetes.io/projected/53d47a1a-f442-4cac-abd3-941e47645e03-kube-api-access-w6dkl\") pod \"network-check-target-8vgqn\" (UID: \"53d47a1a-f442-4cac-abd3-941e47645e03\") " pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:27:21.012792 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:21.012722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5mfk"
Jul 21 17:27:21.031425 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:21.031402    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9vqxq\""
Jul 21 17:27:21.039212 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:21.039190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:27:21.371496 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:21.371464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l5mfk"]
Jul 21 17:27:21.373350 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:21.373324    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1511f43e_7fd8_4f20_86ca_f9eb3c032ac5.slice/crio-55a0311ea4effd29333c055fdbb2f7bb6d50accd3e4261bc394bd5d197e8f9aa WatchSource:0}: Error finding container 55a0311ea4effd29333c055fdbb2f7bb6d50accd3e4261bc394bd5d197e8f9aa: Status 404 returned error can't find the container with id 55a0311ea4effd29333c055fdbb2f7bb6d50accd3e4261bc394bd5d197e8f9aa
Jul 21 17:27:21.590740 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:21.590718    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f6xmv"]
Jul 21 17:27:21.593255 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:21.593225    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f8ea96_7aec_48b2_85ef_6fe96c80ce62.slice/crio-50e5b7b8a833ac5c47b1a01dcc1478e4386af6e972478ff0700da0bcde8e374b WatchSource:0}: Error finding container 50e5b7b8a833ac5c47b1a01dcc1478e4386af6e972478ff0700da0bcde8e374b: Status 404 returned error can't find the container with id 50e5b7b8a833ac5c47b1a01dcc1478e4386af6e972478ff0700da0bcde8e374b
Jul 21 17:27:21.594378 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:21.594338    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8vgqn"]
Jul 21 17:27:21.596255 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:21.596229    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d47a1a_f442_4cac_abd3_941e47645e03.slice/crio-331611ae85eeedcba96745202f91d2bf77819567aa18435d8f8d2a50c59f07c2 WatchSource:0}: Error finding container 331611ae85eeedcba96745202f91d2bf77819567aa18435d8f8d2a50c59f07c2: Status 404 returned error can't find the container with id 331611ae85eeedcba96745202f91d2bf77819567aa18435d8f8d2a50c59f07c2
Jul 21 17:27:22.276201 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:22.276163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8vgqn" event={"ID":"53d47a1a-f442-4cac-abd3-941e47645e03","Type":"ContainerStarted","Data":"331611ae85eeedcba96745202f91d2bf77819567aa18435d8f8d2a50c59f07c2"}
Jul 21 17:27:22.278872 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:22.278827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v92lz" event={"ID":"10f013b0-411e-4cd7-b170-2dd308190cea","Type":"ContainerStarted","Data":"542f94c80feaf7b146b581062b772d955373b33ff6e1c7dc1bb3f91031b32eab"}
Jul 21 17:27:22.280502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:22.280477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6xmv" event={"ID":"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62","Type":"ContainerStarted","Data":"50e5b7b8a833ac5c47b1a01dcc1478e4386af6e972478ff0700da0bcde8e374b"}
Jul 21 17:27:22.281952 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:22.281931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:27:22.284553 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:22.284505    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l5mfk" event={"ID":"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5","Type":"ContainerStarted","Data":"55a0311ea4effd29333c055fdbb2f7bb6d50accd3e4261bc394bd5d197e8f9aa"}
Jul 21 17:27:22.285352 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:22.285329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20be539f-d29f-4ed6-8c74-1635fdea9bd3-metrics-tls\") pod \"dns-default-ccjk8\" (UID: \"20be539f-d29f-4ed6-8c74-1635fdea9bd3\") " pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:27:22.472706 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:22.472673    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s47cz\""
Jul 21 17:27:22.479909 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:22.479875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:27:22.948436 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:22.948349    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ccjk8"]
Jul 21 17:27:23.123003 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:23.122965    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20be539f_d29f_4ed6_8c74_1635fdea9bd3.slice/crio-dd799578086e3b41ba5d18ba501eb8ea05efb2bd0fd3a7e2541594c11466e254 WatchSource:0}: Error finding container dd799578086e3b41ba5d18ba501eb8ea05efb2bd0fd3a7e2541594c11466e254: Status 404 returned error can't find the container with id dd799578086e3b41ba5d18ba501eb8ea05efb2bd0fd3a7e2541594c11466e254
Jul 21 17:27:23.289834 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.289745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ccjk8" event={"ID":"20be539f-d29f-4ed6-8c74-1635fdea9bd3","Type":"ContainerStarted","Data":"dd799578086e3b41ba5d18ba501eb8ea05efb2bd0fd3a7e2541594c11466e254"}
Jul 21 17:27:23.696419 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.696329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:27:23.696419 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.696390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:23.696634 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.696436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:23.697232 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.697207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e56663cb-5de7-4266-b7ca-1aa0bf94928a-service-ca-bundle\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:23.701086 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.701025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") pod \"image-registry-6d55d46bc-pj2hs\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") " pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:27:23.703141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.703094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56663cb-5de7-4266-b7ca-1aa0bf94928a-metrics-certs\") pod \"router-default-9f678b54d-lmmtf\" (UID: \"e56663cb-5de7-4266-b7ca-1aa0bf94928a\") " pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:23.960128 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.960051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c8xpg\""
Jul 21 17:27:23.966831 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.966599    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bwwcr\""
Jul 21 17:27:23.967674 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.967635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:23.974782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:23.974713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:27:24.299041 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:24.298944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6xmv" event={"ID":"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62","Type":"ContainerStarted","Data":"bf0ed6bd3c4ef55d40183f6fe3d229f2756383e6c3f56547d6b5b39180c232ed"}
Jul 21 17:27:24.299594 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:24.299051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6xmv" event={"ID":"a3f8ea96-7aec-48b2-85ef-6fe96c80ce62","Type":"ContainerStarted","Data":"d8d8c3a8a79024350878661f9288f613ccdb59ecb28df60d42e27efd8725aacb"}
Jul 21 17:27:24.317150 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:24.316696    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f6xmv" podStartSLOduration=67.763018103 podStartE2EDuration="1m9.31667587s" totalImagesPullingTime="1.553657767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:24.316521702 +0000 UTC m=+69.875320688" watchObservedRunningTime="2026-07-21 17:27:24.31667587 +0000 UTC m=+69.875474854"
Jul 21 17:27:26.160426 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.160390    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"]
Jul 21 17:27:26.177712 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.177624    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d55d46bc-pj2hs"]
Jul 21 17:27:26.177863 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.177761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.178165 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.178110    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"]
Jul 21 17:27:26.181615 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.181445    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 21 17:27:26.181615 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.181531    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 21 17:27:26.181826 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.181800    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 21 17:27:26.181826 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.181807    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 21 17:27:26.240026 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.239998    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-c4rnj"]
Jul 21 17:27:26.258887 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.258865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-c4rnj"
Jul 21 17:27:26.261499 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.261478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-c4rnj"]
Jul 21 17:27:26.261822 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.261802    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-49bp8\""
Jul 21 17:27:26.262106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.261959    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 21 17:27:26.262604 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.262585    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 21 17:27:26.318963 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.318937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc-tmp\") pod \"klusterlet-addon-workmgr-6f7bf89d48-8z8hm\" (UID: \"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.319114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.318968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f7bf89d48-8z8hm\" (UID: \"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.319114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.318996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snwtz\" (UniqueName: \"kubernetes.io/projected/2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc-kube-api-access-snwtz\") pod \"klusterlet-addon-workmgr-6f7bf89d48-8z8hm\" (UID: \"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.319114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.319050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46v2\" (UniqueName: \"kubernetes.io/projected/6c484675-4f71-4113-a522-e411348e951b-kube-api-access-q46v2\") pod \"downloads-5bc67dcd46-c4rnj\" (UID: \"6c484675-4f71-4113-a522-e411348e951b\") " pod="openshift-console/downloads-5bc67dcd46-c4rnj"
Jul 21 17:27:26.419872 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.419802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc-tmp\") pod \"klusterlet-addon-workmgr-6f7bf89d48-8z8hm\" (UID: \"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.419872 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.419841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f7bf89d48-8z8hm\" (UID: \"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.419872 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.419863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwtz\" (UniqueName: \"kubernetes.io/projected/2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc-kube-api-access-snwtz\") pod \"klusterlet-addon-workmgr-6f7bf89d48-8z8hm\" (UID: \"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.420102 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.419889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46v2\" (UniqueName: \"kubernetes.io/projected/6c484675-4f71-4113-a522-e411348e951b-kube-api-access-q46v2\") pod \"downloads-5bc67dcd46-c4rnj\" (UID: \"6c484675-4f71-4113-a522-e411348e951b\") " pod="openshift-console/downloads-5bc67dcd46-c4rnj"
Jul 21 17:27:26.420264 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.420236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc-tmp\") pod \"klusterlet-addon-workmgr-6f7bf89d48-8z8hm\" (UID: \"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.422874 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.422851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f7bf89d48-8z8hm\" (UID: \"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.431760 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.431737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwtz\" (UniqueName: \"kubernetes.io/projected/2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc-kube-api-access-snwtz\") pod \"klusterlet-addon-workmgr-6f7bf89d48-8z8hm\" (UID: \"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.432061 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.432035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46v2\" (UniqueName: \"kubernetes.io/projected/6c484675-4f71-4113-a522-e411348e951b-kube-api-access-q46v2\") pod \"downloads-5bc67dcd46-c4rnj\" (UID: \"6c484675-4f71-4113-a522-e411348e951b\") " pod="openshift-console/downloads-5bc67dcd46-c4rnj"
Jul 21 17:27:26.488971 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.488948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:26.569878 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:26.569851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-c4rnj"
Jul 21 17:27:29.161109 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.159431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"]
Jul 21 17:27:29.168801 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:29.168704    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbf421c_bb94_4f38_b6e2_f4cd3c5c05fc.slice/crio-fac5810da7acf730576f6afaea5dc685d920837e4daf6fb8b6af40fb2e99820f WatchSource:0}: Error finding container fac5810da7acf730576f6afaea5dc685d920837e4daf6fb8b6af40fb2e99820f: Status 404 returned error can't find the container with id fac5810da7acf730576f6afaea5dc685d920837e4daf6fb8b6af40fb2e99820f
Jul 21 17:27:29.189744 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.189588    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9f678b54d-lmmtf"]
Jul 21 17:27:29.193877 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:29.193850    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56663cb_5de7_4266_b7ca_1aa0bf94928a.slice/crio-d200ca8875e9a67c31c71af8f5cedb723030b38bd6ed9536e6542a10189d116f WatchSource:0}: Error finding container d200ca8875e9a67c31c71af8f5cedb723030b38bd6ed9536e6542a10189d116f: Status 404 returned error can't find the container with id d200ca8875e9a67c31c71af8f5cedb723030b38bd6ed9536e6542a10189d116f
Jul 21 17:27:29.234036 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.234016    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-c4rnj"]
Jul 21 17:27:29.239176 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.239111    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d55d46bc-pj2hs"]
Jul 21 17:27:29.243103 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:29.243074    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c484675_4f71_4113_a522_e411348e951b.slice/crio-ff7b13c398677d361f7010b4093e501a286ff50adba0030f351f043478d07292 WatchSource:0}: Error finding container ff7b13c398677d361f7010b4093e501a286ff50adba0030f351f043478d07292: Status 404 returned error can't find the container with id ff7b13c398677d361f7010b4093e501a286ff50adba0030f351f043478d07292
Jul 21 17:27:29.243614 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:29.243517    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f981f7f_a854_4b87_9c61_8595094ba7cb.slice/crio-25d730c3bb4b412d9c085c241bfd2385e66c4ccd814975c3bf1918c71e0cb681 WatchSource:0}: Error finding container 25d730c3bb4b412d9c085c241bfd2385e66c4ccd814975c3bf1918c71e0cb681: Status 404 returned error can't find the container with id 25d730c3bb4b412d9c085c241bfd2385e66c4ccd814975c3bf1918c71e0cb681
Jul 21 17:27:29.322148 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.322121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs" event={"ID":"3f981f7f-a854-4b87-9c61-8595094ba7cb","Type":"ContainerStarted","Data":"7ff3454dd653a49e0f02a34237053df2879a3c592631a6c5a7a65a49adecf673"}
Jul 21 17:27:29.322259 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.322167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs" event={"ID":"3f981f7f-a854-4b87-9c61-8595094ba7cb","Type":"ContainerStarted","Data":"25d730c3bb4b412d9c085c241bfd2385e66c4ccd814975c3bf1918c71e0cb681"}
Jul 21 17:27:29.322259 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.322248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:27:29.323824 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.323792    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l5mfk" event={"ID":"1511f43e-7fd8-4f20-86ca-f9eb3c032ac5","Type":"ContainerStarted","Data":"9d14b01a97bc4b985395985ed5076fa0f4c669f3c45e1a555406eff3fb050889"}
Jul 21 17:27:29.325007 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.324985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm" event={"ID":"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc","Type":"ContainerStarted","Data":"fac5810da7acf730576f6afaea5dc685d920837e4daf6fb8b6af40fb2e99820f"}
Jul 21 17:27:29.326800 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.326769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9f678b54d-lmmtf" event={"ID":"e56663cb-5de7-4266-b7ca-1aa0bf94928a","Type":"ContainerStarted","Data":"5dbfc3223ee4849c7e8b3e7dc2173f3d0396b00ce881fa111fce055f7cd35e25"}
Jul 21 17:27:29.326900 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.326817    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9f678b54d-lmmtf" event={"ID":"e56663cb-5de7-4266-b7ca-1aa0bf94928a","Type":"ContainerStarted","Data":"d200ca8875e9a67c31c71af8f5cedb723030b38bd6ed9536e6542a10189d116f"}
Jul 21 17:27:29.328803 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.328773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8vgqn" event={"ID":"53d47a1a-f442-4cac-abd3-941e47645e03","Type":"ContainerStarted","Data":"8b285c822e1d43f1d4cc23a3d0cd5d83c4f61ced1dff65624ee1d0c291ff2efc"}
Jul 21 17:27:29.328897 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.328883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:27:29.331016 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.330982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ccjk8" event={"ID":"20be539f-d29f-4ed6-8c74-1635fdea9bd3","Type":"ContainerStarted","Data":"08128cc953337b06de50f00dca171abbbaac878bdac09a7cb540fedd0418f510"}
Jul 21 17:27:29.331111 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.331029    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ccjk8" event={"ID":"20be539f-d29f-4ed6-8c74-1635fdea9bd3","Type":"ContainerStarted","Data":"fe2b02b263be3ea13e892f25af440889c06112ff839a4bba1650e07c761de44c"}
Jul 21 17:27:29.331111 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.331096    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:27:29.332232 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.332211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-c4rnj" event={"ID":"6c484675-4f71-4113-a522-e411348e951b","Type":"ContainerStarted","Data":"ff7b13c398677d361f7010b4093e501a286ff50adba0030f351f043478d07292"}
Jul 21 17:27:29.334303 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.334281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v92lz" event={"ID":"10f013b0-411e-4cd7-b170-2dd308190cea","Type":"ContainerStarted","Data":"043873ca22b86c4c80c66c2c687709d80c7c375f125cfe285b939a496e6c3055"}
Jul 21 17:27:29.348303 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.348260    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs" podStartSLOduration=38.348247135 podStartE2EDuration="38.348247135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:29.347992288 +0000 UTC m=+74.906791292" watchObservedRunningTime="2026-07-21 17:27:29.348247135 +0000 UTC m=+74.907046118"
Jul 21 17:27:29.367956 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.367906    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l5mfk" podStartSLOduration=65.806627345 podStartE2EDuration="1m13.367891844s" totalImagesPullingTime="7.561264499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:29.367807861 +0000 UTC m=+74.926606844" watchObservedRunningTime="2026-07-21 17:27:29.367891844 +0000 UTC m=+74.926690828"
Jul 21 17:27:29.391808 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.391761    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ccjk8" podStartSLOduration=33.600196038 podStartE2EDuration="39.391744401s" totalImagesPullingTime="5.791548363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:29.390525446 +0000 UTC m=+74.949324431" watchObservedRunningTime="2026-07-21 17:27:29.391744401 +0000 UTC m=+74.950543363"
Jul 21 17:27:29.455705 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.455600    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8vgqn" podStartSLOduration=67.098002326 podStartE2EDuration="1m14.455585805s" totalImagesPullingTime="7.357583479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:29.418763353 +0000 UTC m=+74.977562596" watchObservedRunningTime="2026-07-21 17:27:29.455585805 +0000 UTC m=+75.014384789"
Jul 21 17:27:29.485588 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.485535    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v92lz" podStartSLOduration=16.868133785 podStartE2EDuration="32.485519855s" totalImagesPullingTime="15.61738607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:29.480625788 +0000 UTC m=+75.039424772" watchObservedRunningTime="2026-07-21 17:27:29.485519855 +0000 UTC m=+75.044318838"
Jul 21 17:27:29.485876 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.485855    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-9f678b54d-lmmtf" podStartSLOduration=38.485848221 podStartE2EDuration="38.485848221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:29.455550552 +0000 UTC m=+75.014349558" watchObservedRunningTime="2026-07-21 17:27:29.485848221 +0000 UTC m=+75.044647204"
Jul 21 17:27:29.968354 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.968318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:29.971299 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:29.971269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:30.342088 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:30.341597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:30.343339 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:30.343164    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-9f678b54d-lmmtf"
Jul 21 17:27:33.743970 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.743897    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-d759z"]
Jul 21 17:27:33.747899 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.747877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.751062 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.751043    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 21 17:27:33.752837 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.752813    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 21 17:27:33.752837 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.752830    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 21 17:27:33.752996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.752813    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 21 17:27:33.752996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.752834    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-r2f6r\""
Jul 21 17:27:33.753096 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.752813    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 21 17:27:33.756939 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.756850    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-d759z"]
Jul 21 17:27:33.885505 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.885468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.885701 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.885528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.885701 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.885604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.885820 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.885711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq6pq\" (UniqueName: \"kubernetes.io/projected/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-kube-api-access-wq6pq\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.986734 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.986697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.986883 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.986753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6pq\" (UniqueName: \"kubernetes.io/projected/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-kube-api-access-wq6pq\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.986883 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.986785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.986883 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.986817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.987476 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.987447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.989528 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.989504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.989633 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.989556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:33.999334 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:33.999274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6pq\" (UniqueName: \"kubernetes.io/projected/cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9-kube-api-access-wq6pq\") pod \"prometheus-operator-5945b5cfd-d759z\" (UID: \"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:34.060591 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:34.060564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z"
Jul 21 17:27:34.209269 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:34.209093    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-d759z"]
Jul 21 17:27:34.212030 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:34.211998    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0ae0ca_320d_48bc_9ba8_b5f68ef6b5a9.slice/crio-788772be6500182e6a32faa649e517d05b3c900416914c859cb8c8d2167de68f WatchSource:0}: Error finding container 788772be6500182e6a32faa649e517d05b3c900416914c859cb8c8d2167de68f: Status 404 returned error can't find the container with id 788772be6500182e6a32faa649e517d05b3c900416914c859cb8c8d2167de68f
Jul 21 17:27:34.365539 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:34.365496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm" event={"ID":"2fbf421c-bb94-4f38-b6e2-f4cd3c5c05fc","Type":"ContainerStarted","Data":"ab4ae34c1494bbfb4b44eb15d1e1cccdb93906ebc5f09516599383714dab5f1d"}
Jul 21 17:27:34.365772 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:34.365754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:34.366862 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:34.366837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z" event={"ID":"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9","Type":"ContainerStarted","Data":"788772be6500182e6a32faa649e517d05b3c900416914c859cb8c8d2167de68f"}
Jul 21 17:27:34.367672 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:34.367634    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm"
Jul 21 17:27:34.384507 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:34.384465    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f7bf89d48-8z8hm" podStartSLOduration=4.115706308 podStartE2EDuration="8.384451409s" totalImagesPullingTime="4.268745101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:34.383339 +0000 UTC m=+79.942137985" watchObservedRunningTime="2026-07-21 17:27:34.384451409 +0000 UTC m=+79.943250392"
Jul 21 17:27:36.376987 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:36.376947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z" event={"ID":"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9","Type":"ContainerStarted","Data":"9336ee76ca9589e647fb99bd92b75b798850fac6a240b91ac5a8689735203018"}
Jul 21 17:27:36.377473 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:36.377043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z" event={"ID":"cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9","Type":"ContainerStarted","Data":"5d4b820afc6e5804f7daa59d9ca06f5a8faeb96273438a5ef23175ffe30b85c3"}
Jul 21 17:27:36.398114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:36.398069    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-d759z" podStartSLOduration=2.142651739 podStartE2EDuration="3.398056483s" totalImagesPullingTime="1.255404744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:36.397913497 +0000 UTC m=+81.956712485" watchObservedRunningTime="2026-07-21 17:27:36.398056483 +0000 UTC m=+81.956855466"
Jul 21 17:27:38.614005 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.613971    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mhjhd"]
Jul 21 17:27:38.617174 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.617141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.621723 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.621578    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 21 17:27:38.621824 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.621738    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 21 17:27:38.621824 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.621744    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tzftz\""
Jul 21 17:27:38.621919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.621857    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 21 17:27:38.728084 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.728056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-tls\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.728260 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.728098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b71adf0f-c005-45ea-a6a9-b4684095d877-root\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.728260 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.728116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71adf0f-c005-45ea-a6a9-b4684095d877-sys\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.728260 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.728179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.728260 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.728245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-wtmp\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.728479 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.728265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.728479 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.728300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-textfile\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.728479 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.728336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b71adf0f-c005-45ea-a6a9-b4684095d877-metrics-client-ca\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.728479 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.728383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zvg\" (UniqueName: \"kubernetes.io/projected/b71adf0f-c005-45ea-a6a9-b4684095d877-kube-api-access-k8zvg\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.829642 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.829611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-textfile\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.829827 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.829686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b71adf0f-c005-45ea-a6a9-b4684095d877-metrics-client-ca\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.829827 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.829720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zvg\" (UniqueName: \"kubernetes.io/projected/b71adf0f-c005-45ea-a6a9-b4684095d877-kube-api-access-k8zvg\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.829940 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.829867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-tls\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.829999 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.829914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b71adf0f-c005-45ea-a6a9-b4684095d877-root\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.829999 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.829969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71adf0f-c005-45ea-a6a9-b4684095d877-sys\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.829999 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.829983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-textfile\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.830150 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.830026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.830150 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.830044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b71adf0f-c005-45ea-a6a9-b4684095d877-root\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.830150 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.830064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-wtmp\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.830150 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.830090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.830351 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.830209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b71adf0f-c005-45ea-a6a9-b4684095d877-metrics-client-ca\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.830351 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.830313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-wtmp\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.830489 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.830366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71adf0f-c005-45ea-a6a9-b4684095d877-sys\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.830546 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.830509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.832519 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.832498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.832519 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.832519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b71adf0f-c005-45ea-a6a9-b4684095d877-node-exporter-tls\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.840210 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.840186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zvg\" (UniqueName: \"kubernetes.io/projected/b71adf0f-c005-45ea-a6a9-b4684095d877-kube-api-access-k8zvg\") pod \"node-exporter-mhjhd\" (UID: \"b71adf0f-c005-45ea-a6a9-b4684095d877\") " pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:38.928385 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:38.928317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhjhd"
Jul 21 17:27:39.344393 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.344312    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ccjk8"
Jul 21 17:27:39.414767 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.414722    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:27:39.419312 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.419292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.425283 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.425257    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 21 17:27:39.425445 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.425427    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 21 17:27:39.425637 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.425616    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 21 17:27:39.425741 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.425704    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 21 17:27:39.426271 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.426188    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 21 17:27:39.426452 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.426412    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 21 17:27:39.426787 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.426770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8nw54\""
Jul 21 17:27:39.427160 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.427003    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 21 17:27:39.427160 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.427100    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 21 17:27:39.427305 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.427100    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 21 17:27:39.445233 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.445198    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:27:39.535171 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535171 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535422 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535422 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535422 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-volume\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535422 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535631 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535631 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4l7l\" (UniqueName: \"kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-kube-api-access-z4l7l\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535631 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535631 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535631 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-out\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535631 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.535631 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.535615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-web-config\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.636963 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.636922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.636963 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.636968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-web-config\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-volume\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4l7l\" (UniqueName: \"kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-kube-api-access-z4l7l\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.637513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.637465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-out\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.638313 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.638287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.638935 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.638911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.639095 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.639079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.640769 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.640744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-out\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.641144 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.641058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.641144 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.641073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.641392 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.641306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.642009 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.641877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-web-config\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.642271 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.642248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-volume\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.642497 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.642474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.642668 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.642616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.643633 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.643574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.647163 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.647144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4l7l\" (UniqueName: \"kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-kube-api-access-z4l7l\") pod \"alertmanager-main-0\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:39.732305 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:39.732269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:40.404813 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.404784    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-644cfc6bf6-kzzhr"]
Jul 21 17:27:40.416207 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.416186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.419546 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.419520    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 21 17:27:40.419692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.419635    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 21 17:27:40.421013 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.420990    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9svqm\""
Jul 21 17:27:40.421110 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.421077    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 21 17:27:40.421178 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.421098    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 21 17:27:40.421178 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.421114    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 21 17:27:40.426560 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.426527    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 21 17:27:40.429320 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.429279    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644cfc6bf6-kzzhr"]
Jul 21 17:27:40.545545 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.545511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-oauth-serving-cert\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.545746 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.545562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-oauth-config\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.545746 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.545670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-trusted-ca-bundle\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.545746 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.545703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jhbr\" (UniqueName: \"kubernetes.io/projected/df68ad25-3a6d-4163-8d2d-25e430bf7b60-kube-api-access-9jhbr\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.545746 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.545738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-service-ca\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.545944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.545797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-serving-cert\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.545944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.545814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-config\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.646386 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.646351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-trusted-ca-bundle\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.646901 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.646397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jhbr\" (UniqueName: \"kubernetes.io/projected/df68ad25-3a6d-4163-8d2d-25e430bf7b60-kube-api-access-9jhbr\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.646901 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.646435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-service-ca\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.646901 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.646495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-serving-cert\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.646901 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.646518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-config\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.646901 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.646558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-oauth-serving-cert\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.646901 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.646592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-oauth-config\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.647315 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.647288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-service-ca\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.647373 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.647358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-oauth-serving-cert\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.647422 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.647404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-config\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.647472 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.647405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-trusted-ca-bundle\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.649203 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.649181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-oauth-config\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.649408 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.649387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-serving-cert\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.657834 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.657778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jhbr\" (UniqueName: \"kubernetes.io/projected/df68ad25-3a6d-4163-8d2d-25e430bf7b60-kube-api-access-9jhbr\") pod \"console-644cfc6bf6-kzzhr\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") " pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:40.727668 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:40.727621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:27:42.933990 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:42.932759    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"]
Jul 21 17:27:43.040470 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.040434    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"]
Jul 21 17:27:43.040675 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.040574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"
Jul 21 17:27:43.043731 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.043673    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7bm7l\""
Jul 21 17:27:43.043731 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.043682    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 21 17:27:43.168558 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.168522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9ksv\" (UniqueName: \"kubernetes.io/projected/76603123-4844-43ff-bedf-472ce2c72a9c-kube-api-access-r9ksv\") pod \"monitoring-plugin-7b7b7bc8bf-crfrn\" (UID: \"76603123-4844-43ff-bedf-472ce2c72a9c\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"
Jul 21 17:27:43.168774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.168613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/76603123-4844-43ff-bedf-472ce2c72a9c-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-crfrn\" (UID: \"76603123-4844-43ff-bedf-472ce2c72a9c\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"
Jul 21 17:27:43.269808 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.269637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9ksv\" (UniqueName: \"kubernetes.io/projected/76603123-4844-43ff-bedf-472ce2c72a9c-kube-api-access-r9ksv\") pod \"monitoring-plugin-7b7b7bc8bf-crfrn\" (UID: \"76603123-4844-43ff-bedf-472ce2c72a9c\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"
Jul 21 17:27:43.269808 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.269723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/76603123-4844-43ff-bedf-472ce2c72a9c-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-crfrn\" (UID: \"76603123-4844-43ff-bedf-472ce2c72a9c\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"
Jul 21 17:27:43.272543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.272523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/76603123-4844-43ff-bedf-472ce2c72a9c-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-crfrn\" (UID: \"76603123-4844-43ff-bedf-472ce2c72a9c\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"
Jul 21 17:27:43.285325 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.285298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9ksv\" (UniqueName: \"kubernetes.io/projected/76603123-4844-43ff-bedf-472ce2c72a9c-kube-api-access-r9ksv\") pod \"monitoring-plugin-7b7b7bc8bf-crfrn\" (UID: \"76603123-4844-43ff-bedf-472ce2c72a9c\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"
Jul 21 17:27:43.352350 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:43.352309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"
Jul 21 17:27:44.214633 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:44.214599    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-644cfc6bf6-kzzhr"]
Jul 21 17:27:46.020206 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:46.020169    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71adf0f_c005_45ea_a6a9_b4684095d877.slice/crio-101702d80beeaf472f4d4c8f4ad9e8d0696e6145c0f344eaf0794152e7403fd5 WatchSource:0}: Error finding container 101702d80beeaf472f4d4c8f4ad9e8d0696e6145c0f344eaf0794152e7403fd5: Status 404 returned error can't find the container with id 101702d80beeaf472f4d4c8f4ad9e8d0696e6145c0f344eaf0794152e7403fd5
Jul 21 17:27:46.174197 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:46.174157    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:27:46.174636 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:46.174615    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fa2412_8f55_4919_8be1_3d703b6dd23c.slice/crio-8dad662f7c2fcd94a85876dafadc7263779541121c44f540c38882a257c1efd9 WatchSource:0}: Error finding container 8dad662f7c2fcd94a85876dafadc7263779541121c44f540c38882a257c1efd9: Status 404 returned error can't find the container with id 8dad662f7c2fcd94a85876dafadc7263779541121c44f540c38882a257c1efd9
Jul 21 17:27:46.385503 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:46.385481    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-644cfc6bf6-kzzhr"]
Jul 21 17:27:46.387630 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:46.387607    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"]
Jul 21 17:27:46.388111 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:46.388088    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf68ad25_3a6d_4163_8d2d_25e430bf7b60.slice/crio-fce5f36c8f3c5003a9e209a2fea04f1fb0cb8c56dbd57fc69ef7f9971b8a909b WatchSource:0}: Error finding container fce5f36c8f3c5003a9e209a2fea04f1fb0cb8c56dbd57fc69ef7f9971b8a909b: Status 404 returned error can't find the container with id fce5f36c8f3c5003a9e209a2fea04f1fb0cb8c56dbd57fc69ef7f9971b8a909b
Jul 21 17:27:46.390204 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:27:46.390179    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76603123_4844_43ff_bedf_472ce2c72a9c.slice/crio-916d03f6fc5d2bb9a72c44c8a06c3e5dfc6ccd0b01d22a542f110d54e8221ef8 WatchSource:0}: Error finding container 916d03f6fc5d2bb9a72c44c8a06c3e5dfc6ccd0b01d22a542f110d54e8221ef8: Status 404 returned error can't find the container with id 916d03f6fc5d2bb9a72c44c8a06c3e5dfc6ccd0b01d22a542f110d54e8221ef8
Jul 21 17:27:46.422612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:46.422559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn" event={"ID":"76603123-4844-43ff-bedf-472ce2c72a9c","Type":"ContainerStarted","Data":"916d03f6fc5d2bb9a72c44c8a06c3e5dfc6ccd0b01d22a542f110d54e8221ef8"}
Jul 21 17:27:46.423901 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:46.423877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhjhd" event={"ID":"b71adf0f-c005-45ea-a6a9-b4684095d877","Type":"ContainerStarted","Data":"101702d80beeaf472f4d4c8f4ad9e8d0696e6145c0f344eaf0794152e7403fd5"}
Jul 21 17:27:46.425340 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:46.425297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644cfc6bf6-kzzhr" event={"ID":"df68ad25-3a6d-4163-8d2d-25e430bf7b60","Type":"ContainerStarted","Data":"fce5f36c8f3c5003a9e209a2fea04f1fb0cb8c56dbd57fc69ef7f9971b8a909b"}
Jul 21 17:27:46.426994 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:46.426954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerStarted","Data":"8dad662f7c2fcd94a85876dafadc7263779541121c44f540c38882a257c1efd9"}
Jul 21 17:27:47.440701 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:47.440625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-c4rnj" event={"ID":"6c484675-4f71-4113-a522-e411348e951b","Type":"ContainerStarted","Data":"b81f38abf6fc92a52763356cb4cb7c1231598f0d8bff94823c0dddb991f0c659"}
Jul 21 17:27:47.471809 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:47.471746    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-c4rnj" podStartSLOduration=4.340712519 podStartE2EDuration="21.471724947s" totalImagesPullingTime="17.131012428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:47.469841711 +0000 UTC m=+93.028640708" watchObservedRunningTime="2026-07-21 17:27:47.471724947 +0000 UTC m=+93.030523933"
Jul 21 17:27:48.449312 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:48.449276    2579 generic.go:356] "Generic (PLEG): container finished" podID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerID="c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee" exitCode=0
Jul 21 17:27:48.449780 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:48.449444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerDied","Data":"c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee"}
Jul 21 17:27:48.454478 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:48.454449    2579 generic.go:356] "Generic (PLEG): container finished" podID="b71adf0f-c005-45ea-a6a9-b4684095d877" containerID="8b0e0a0c0972454fe898e0396aa777ec31860a0d85387c6bbfee464eeee19797" exitCode=0
Jul 21 17:27:48.455846 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:48.455810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhjhd" event={"ID":"b71adf0f-c005-45ea-a6a9-b4684095d877","Type":"ContainerDied","Data":"8b0e0a0c0972454fe898e0396aa777ec31860a0d85387c6bbfee464eeee19797"}
Jul 21 17:27:48.455944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:48.455903    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-c4rnj"
Jul 21 17:27:48.469139 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:48.469095    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-c4rnj"
Jul 21 17:27:49.330573 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:49.330431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:27:49.462079 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:49.462031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhjhd" event={"ID":"b71adf0f-c005-45ea-a6a9-b4684095d877","Type":"ContainerStarted","Data":"77d45413d009cd6025976f4592619de07f6e3189dbb3daab48bec30b506bbf48"}
Jul 21 17:27:49.462531 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:49.462095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhjhd" event={"ID":"b71adf0f-c005-45ea-a6a9-b4684095d877","Type":"ContainerStarted","Data":"564e66ec8f9c7d33c0e7f2f173b1e33bd3af561e6ffbc108a517260c9c97a462"}
Jul 21 17:27:49.486733 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:49.486678    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mhjhd" podStartSLOduration=9.354696541 podStartE2EDuration="11.486644553s" totalImagesPullingTime="2.00614829s" totalInitContainerRuntime="125.799722ms" isStatefulPod=true podCreationTimestamp="2026-07-21 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:49.484929089 +0000 UTC m=+95.043728087" watchObservedRunningTime="2026-07-21 17:27:49.486644553 +0000 UTC m=+95.045443539"
Jul 21 17:27:52.478500 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:52.478462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn" event={"ID":"76603123-4844-43ff-bedf-472ce2c72a9c","Type":"ContainerStarted","Data":"3c78e7a9fabccacaed280eb1d0605a03e4fb0568f7cee8fd1abbb0c8818f941c"}
Jul 21 17:27:52.480305 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:52.480272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644cfc6bf6-kzzhr" event={"ID":"df68ad25-3a6d-4163-8d2d-25e430bf7b60","Type":"ContainerStarted","Data":"5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c"}
Jul 21 17:27:52.482769 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:52.482738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerStarted","Data":"68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2"}
Jul 21 17:27:52.482863 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:52.482795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerStarted","Data":"42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b"}
Jul 21 17:27:52.488644 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:52.488601    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"
Jul 21 17:27:52.494191 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:52.494167    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn"
Jul 21 17:27:52.511506 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:52.511463    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-644cfc6bf6-kzzhr" podStartSLOduration=7.112895371 podStartE2EDuration="12.511447501s" totalImagesPullingTime="5.39855213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:52.510802762 +0000 UTC m=+98.069601746" watchObservedRunningTime="2026-07-21 17:27:52.511447501 +0000 UTC m=+98.070246484"
Jul 21 17:27:52.531747 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:52.531703    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-crfrn" podStartSLOduration=5.142778756 podStartE2EDuration="10.531684334s" totalImagesPullingTime="5.388905578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:52.529590925 +0000 UTC m=+98.088389911" watchObservedRunningTime="2026-07-21 17:27:52.531684334 +0000 UTC m=+98.090483318"
Jul 21 17:27:53.491061 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:53.491006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerStarted","Data":"b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b"}
Jul 21 17:27:53.491474 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:53.491115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerStarted","Data":"5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4"}
Jul 21 17:27:53.491474 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:53.491163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerStarted","Data":"14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678"}
Jul 21 17:27:54.345360 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.345320    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs" podUID="3f981f7f-a854-4b87-9c61-8595094ba7cb" containerName="registry" containerID="cri-o://7ff3454dd653a49e0f02a34237053df2879a3c592631a6c5a7a65a49adecf673" gracePeriod=30
Jul 21 17:27:54.498021 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.497967    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f981f7f-a854-4b87-9c61-8595094ba7cb" containerID="7ff3454dd653a49e0f02a34237053df2879a3c592631a6c5a7a65a49adecf673" exitCode=0
Jul 21 17:27:54.498410 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.498061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs" event={"ID":"3f981f7f-a854-4b87-9c61-8595094ba7cb","Type":"ContainerDied","Data":"7ff3454dd653a49e0f02a34237053df2879a3c592631a6c5a7a65a49adecf673"}
Jul 21 17:27:54.502421 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.502389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerStarted","Data":"4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9"}
Jul 21 17:27:54.542260 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.542209    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.531740132 podStartE2EDuration="15.542189299s" totalImagesPullingTime="6.987412607s" totalInitContainerRuntime="23.03656ms" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:54.540074704 +0000 UTC m=+100.098873712" watchObservedRunningTime="2026-07-21 17:27:54.542189299 +0000 UTC m=+100.100988284"
Jul 21 17:27:54.649012 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.648983    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:27:54.687141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.687114    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-installation-pull-secrets\") pod \"3f981f7f-a854-4b87-9c61-8595094ba7cb\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") "
Jul 21 17:27:54.687290 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.687189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") pod \"3f981f7f-a854-4b87-9c61-8595094ba7cb\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") "
Jul 21 17:27:54.687290 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.687222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-image-registry-private-configuration\") pod \"3f981f7f-a854-4b87-9c61-8595094ba7cb\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") "
Jul 21 17:27:54.687290 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.687251    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgr69\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-kube-api-access-vgr69\") pod \"3f981f7f-a854-4b87-9c61-8595094ba7cb\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") "
Jul 21 17:27:54.687290 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.687277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-trusted-ca\") pod \"3f981f7f-a854-4b87-9c61-8595094ba7cb\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") "
Jul 21 17:27:54.687502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.687309    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f981f7f-a854-4b87-9c61-8595094ba7cb-ca-trust-extracted\") pod \"3f981f7f-a854-4b87-9c61-8595094ba7cb\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") "
Jul 21 17:27:54.687502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.687344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-bound-sa-token\") pod \"3f981f7f-a854-4b87-9c61-8595094ba7cb\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") "
Jul 21 17:27:54.687502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.687381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-certificates\") pod \"3f981f7f-a854-4b87-9c61-8595094ba7cb\" (UID: \"3f981f7f-a854-4b87-9c61-8595094ba7cb\") "
Jul 21 17:27:54.688200 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.688010    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3f981f7f-a854-4b87-9c61-8595094ba7cb" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:27:54.688593 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.688566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3f981f7f-a854-4b87-9c61-8595094ba7cb" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:27:54.690751 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.690721    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3f981f7f-a854-4b87-9c61-8595094ba7cb" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:27:54.691146 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.691119    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3f981f7f-a854-4b87-9c61-8595094ba7cb" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:27:54.691238 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.691208    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3f981f7f-a854-4b87-9c61-8595094ba7cb" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:27:54.691963 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.691880    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3f981f7f-a854-4b87-9c61-8595094ba7cb" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:27:54.692544 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.692517    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-kube-api-access-vgr69" (OuterVolumeSpecName: "kube-api-access-vgr69") pod "3f981f7f-a854-4b87-9c61-8595094ba7cb" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb"). InnerVolumeSpecName "kube-api-access-vgr69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:27:54.698277 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.698247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f981f7f-a854-4b87-9c61-8595094ba7cb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3f981f7f-a854-4b87-9c61-8595094ba7cb" (UID: "3f981f7f-a854-4b87-9c61-8595094ba7cb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:27:54.787995 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.787932    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-image-registry-private-configuration\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:27:54.787995 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.787959    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgr69\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-kube-api-access-vgr69\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:27:54.787995 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.787970    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-trusted-ca\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:27:54.787995 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.787978    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f981f7f-a854-4b87-9c61-8595094ba7cb-ca-trust-extracted\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:27:54.787995 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.787987    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-bound-sa-token\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:27:54.787995 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.787994    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-certificates\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:27:54.788299 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.788003    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f981f7f-a854-4b87-9c61-8595094ba7cb-installation-pull-secrets\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:27:54.788299 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:54.788011    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f981f7f-a854-4b87-9c61-8595094ba7cb-registry-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:27:55.508336 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:55.508254    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs"
Jul 21 17:27:55.508336 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:55.508274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d55d46bc-pj2hs" event={"ID":"3f981f7f-a854-4b87-9c61-8595094ba7cb","Type":"ContainerDied","Data":"25d730c3bb4b412d9c085c241bfd2385e66c4ccd814975c3bf1918c71e0cb681"}
Jul 21 17:27:55.508336 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:55.508331    2579 scope.go:117] "RemoveContainer" containerID="7ff3454dd653a49e0f02a34237053df2879a3c592631a6c5a7a65a49adecf673"
Jul 21 17:27:55.532792 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:55.532768    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d55d46bc-pj2hs"]
Jul 21 17:27:55.541466 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:55.541443    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d55d46bc-pj2hs"]
Jul 21 17:27:57.003193 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:27:57.003156    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f981f7f-a854-4b87-9c61-8595094ba7cb" path="/var/lib/kubelet/pods/3f981f7f-a854-4b87-9c61-8595094ba7cb/volumes"
Jul 21 17:28:00.343960 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:00.343928    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8vgqn"
Jul 21 17:28:00.729392 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:00.728437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:28:12.571087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:12.571056    2579 generic.go:356] "Generic (PLEG): container finished" podID="8677903c-34d1-41e8-b0c7-be1e3188b4ce" containerID="732f95a4adb0b2640122a8f090ffef07a2510332d533d4b152241d103ffa473f" exitCode=0
Jul 21 17:28:12.571492 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:12.571127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-x2sft" event={"ID":"8677903c-34d1-41e8-b0c7-be1e3188b4ce","Type":"ContainerDied","Data":"732f95a4adb0b2640122a8f090ffef07a2510332d533d4b152241d103ffa473f"}
Jul 21 17:28:12.571492 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:12.571429    2579 scope.go:117] "RemoveContainer" containerID="732f95a4adb0b2640122a8f090ffef07a2510332d533d4b152241d103ffa473f"
Jul 21 17:28:13.576719 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:13.576682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-x2sft" event={"ID":"8677903c-34d1-41e8-b0c7-be1e3188b4ce","Type":"ContainerStarted","Data":"da50b067894b5b4603a9b3b5d6d7185b9cd4b4d84f76ab72cb5bd02d8b5bb889"}
Jul 21 17:28:17.513409 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.513350    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-644cfc6bf6-kzzhr" podUID="df68ad25-3a6d-4163-8d2d-25e430bf7b60" containerName="console" containerID="cri-o://5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c" gracePeriod=15
Jul 21 17:28:17.782672 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.782591    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:28:17.872155 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872122    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-trusted-ca-bundle\") pod \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") "
Jul 21 17:28:17.872155 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-config\") pod \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") "
Jul 21 17:28:17.872376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872183    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-serving-cert\") pod \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") "
Jul 21 17:28:17.872376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872204    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-service-ca\") pod \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") "
Jul 21 17:28:17.872376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-oauth-config\") pod \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") "
Jul 21 17:28:17.872376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872249    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-oauth-serving-cert\") pod \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") "
Jul 21 17:28:17.872376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872265    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jhbr\" (UniqueName: \"kubernetes.io/projected/df68ad25-3a6d-4163-8d2d-25e430bf7b60-kube-api-access-9jhbr\") pod \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\" (UID: \"df68ad25-3a6d-4163-8d2d-25e430bf7b60\") "
Jul 21 17:28:17.872636 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872601    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "df68ad25-3a6d-4163-8d2d-25e430bf7b60" (UID: "df68ad25-3a6d-4163-8d2d-25e430bf7b60"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:17.872778 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872672    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-service-ca" (OuterVolumeSpecName: "service-ca") pod "df68ad25-3a6d-4163-8d2d-25e430bf7b60" (UID: "df68ad25-3a6d-4163-8d2d-25e430bf7b60"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:17.872851 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872795    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-config" (OuterVolumeSpecName: "console-config") pod "df68ad25-3a6d-4163-8d2d-25e430bf7b60" (UID: "df68ad25-3a6d-4163-8d2d-25e430bf7b60"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:17.872906 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.872872    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "df68ad25-3a6d-4163-8d2d-25e430bf7b60" (UID: "df68ad25-3a6d-4163-8d2d-25e430bf7b60"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:17.874472 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.874445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "df68ad25-3a6d-4163-8d2d-25e430bf7b60" (UID: "df68ad25-3a6d-4163-8d2d-25e430bf7b60"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:17.874472 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.874458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df68ad25-3a6d-4163-8d2d-25e430bf7b60-kube-api-access-9jhbr" (OuterVolumeSpecName: "kube-api-access-9jhbr") pod "df68ad25-3a6d-4163-8d2d-25e430bf7b60" (UID: "df68ad25-3a6d-4163-8d2d-25e430bf7b60"). InnerVolumeSpecName "kube-api-access-9jhbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:28:17.874472 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.874458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "df68ad25-3a6d-4163-8d2d-25e430bf7b60" (UID: "df68ad25-3a6d-4163-8d2d-25e430bf7b60"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:17.972955 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.972925    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-serving-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:28:17.972955 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.972950    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-service-ca\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:28:17.972955 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.972959    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-oauth-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:28:17.973166 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.972968    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-oauth-serving-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:28:17.973166 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.972976    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jhbr\" (UniqueName: \"kubernetes.io/projected/df68ad25-3a6d-4163-8d2d-25e430bf7b60-kube-api-access-9jhbr\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:28:17.973166 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.972986    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-trusted-ca-bundle\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:28:17.973166 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:17.972995    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df68ad25-3a6d-4163-8d2d-25e430bf7b60-console-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:28:18.598222 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:18.598179    2579 generic.go:356] "Generic (PLEG): container finished" podID="df68ad25-3a6d-4163-8d2d-25e430bf7b60" containerID="5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c" exitCode=0
Jul 21 17:28:18.598692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:18.598232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644cfc6bf6-kzzhr" event={"ID":"df68ad25-3a6d-4163-8d2d-25e430bf7b60","Type":"ContainerDied","Data":"5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c"}
Jul 21 17:28:18.598692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:18.598303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644cfc6bf6-kzzhr" event={"ID":"df68ad25-3a6d-4163-8d2d-25e430bf7b60","Type":"ContainerDied","Data":"fce5f36c8f3c5003a9e209a2fea04f1fb0cb8c56dbd57fc69ef7f9971b8a909b"}
Jul 21 17:28:18.598692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:18.598340    2579 scope.go:117] "RemoveContainer" containerID="5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c"
Jul 21 17:28:18.598692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:18.598531    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644cfc6bf6-kzzhr"
Jul 21 17:28:18.606995 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:18.606693    2579 scope.go:117] "RemoveContainer" containerID="5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c"
Jul 21 17:28:18.607070 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:28:18.607022    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c\": container with ID starting with 5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c not found: ID does not exist" containerID="5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c"
Jul 21 17:28:18.607122 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:18.607061    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c"} err="failed to get container status \"5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c\": rpc error: code = NotFound desc = could not find container \"5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c\": container with ID starting with 5a313e36cbd61e91ec697637fbff043526b7e0f6f46b2a2c8b2a8a672f6b6e5c not found: ID does not exist"
Jul 21 17:28:18.621054 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:18.621029    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-644cfc6bf6-kzzhr"]
Jul 21 17:28:18.624786 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:18.624766    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-644cfc6bf6-kzzhr"]
Jul 21 17:28:19.003384 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:19.003350    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df68ad25-3a6d-4163-8d2d-25e430bf7b60" path="/var/lib/kubelet/pods/df68ad25-3a6d-4163-8d2d-25e430bf7b60/volumes"
Jul 21 17:28:21.611333 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:21.611303    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f5d4bf0-deac-486f-a3dd-a5ec092c342d" containerID="815fcb52488ea7be7780f38bbd41a2b72a055b0f459ff7b5dd92bb612cf56ffa" exitCode=0
Jul 21 17:28:21.611755 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:21.611374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj" event={"ID":"3f5d4bf0-deac-486f-a3dd-a5ec092c342d","Type":"ContainerDied","Data":"815fcb52488ea7be7780f38bbd41a2b72a055b0f459ff7b5dd92bb612cf56ffa"}
Jul 21 17:28:21.611755 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:21.611713    2579 scope.go:117] "RemoveContainer" containerID="815fcb52488ea7be7780f38bbd41a2b72a055b0f459ff7b5dd92bb612cf56ffa"
Jul 21 17:28:22.616592 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:22.616560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-prmlj" event={"ID":"3f5d4bf0-deac-486f-a3dd-a5ec092c342d","Type":"ContainerStarted","Data":"653e81358e75cfb3b2d89a23f8108c2917d9fe2a41e13226f023b96860f16901"}
Jul 21 17:28:58.692337 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.692307    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-557b5d69d4-jzgn8"]
Jul 21 17:28:58.692762 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.692572    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f981f7f-a854-4b87-9c61-8595094ba7cb" containerName="registry"
Jul 21 17:28:58.692762 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.692584    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f981f7f-a854-4b87-9c61-8595094ba7cb" containerName="registry"
Jul 21 17:28:58.692762 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.692600    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df68ad25-3a6d-4163-8d2d-25e430bf7b60" containerName="console"
Jul 21 17:28:58.692762 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.692605    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df68ad25-3a6d-4163-8d2d-25e430bf7b60" containerName="console"
Jul 21 17:28:58.692762 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.692670    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f981f7f-a854-4b87-9c61-8595094ba7cb" containerName="registry"
Jul 21 17:28:58.692762 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.692678    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df68ad25-3a6d-4163-8d2d-25e430bf7b60" containerName="console"
Jul 21 17:28:58.695337 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.695320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.698261 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.698240    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 21 17:28:58.698449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.698430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 21 17:28:58.698515 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.698502    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 21 17:28:58.698570 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.698522    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 21 17:28:58.698570 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.698504    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 21 17:28:58.698570 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.698537    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9svqm\""
Jul 21 17:28:58.703548 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.703526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 21 17:28:58.705089 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.705068    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-557b5d69d4-jzgn8"]
Jul 21 17:28:58.883543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.883509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-config\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.883543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.883539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-service-ca\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.883742 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.883559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-oauth-config\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.883742 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.883666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4pk\" (UniqueName: \"kubernetes.io/projected/3f754a3b-88f2-4c0a-81b4-873ba820c568-kube-api-access-st4pk\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.883742 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.883716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-trusted-ca-bundle\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.883834 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.883742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-serving-cert\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.883834 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.883761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-oauth-serving-cert\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.893550 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.893524    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:28:58.893840 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.893818    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="alertmanager" containerID="cri-o://42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b" gracePeriod=120
Jul 21 17:28:58.893904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.893876    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy-metric" containerID="cri-o://b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b" gracePeriod=120
Jul 21 17:28:58.893953 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.893915    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy" containerID="cri-o://5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4" gracePeriod=120
Jul 21 17:28:58.894003 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.893889    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy-web" containerID="cri-o://14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678" gracePeriod=120
Jul 21 17:28:58.894051 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.893971    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="prom-label-proxy" containerID="cri-o://4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9" gracePeriod=120
Jul 21 17:28:58.894051 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.893985    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="config-reloader" containerID="cri-o://68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2" gracePeriod=120
Jul 21 17:28:58.984723 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.984626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4pk\" (UniqueName: \"kubernetes.io/projected/3f754a3b-88f2-4c0a-81b4-873ba820c568-kube-api-access-st4pk\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.984868 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.984784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-trusted-ca-bundle\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.984868 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.984831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-serving-cert\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.984980 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.984874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-oauth-serving-cert\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.984980 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.984927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-config\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.984980 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.984950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-service-ca\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.985115 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.984981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-oauth-config\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.985597 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.985554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-oauth-serving-cert\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.985735 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.985712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-config\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.985786 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.985740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-service-ca\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.985855 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.985839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-trusted-ca-bundle\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.987163 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.987146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-oauth-config\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.987260 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.987245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-serving-cert\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:58.992919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:58.992901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4pk\" (UniqueName: \"kubernetes.io/projected/3f754a3b-88f2-4c0a-81b4-873ba820c568-kube-api-access-st4pk\") pod \"console-557b5d69d4-jzgn8\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") " pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:59.005187 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.005168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:28:59.140543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.140520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-557b5d69d4-jzgn8"]
Jul 21 17:28:59.142853 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:28:59.142829    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f754a3b_88f2_4c0a_81b4_873ba820c568.slice/crio-c3809a8555b4ec9908bbfbf78a2a66fe08a661079e649a1095a114f61e6d60e6 WatchSource:0}: Error finding container c3809a8555b4ec9908bbfbf78a2a66fe08a661079e649a1095a114f61e6d60e6: Status 404 returned error can't find the container with id c3809a8555b4ec9908bbfbf78a2a66fe08a661079e649a1095a114f61e6d60e6
Jul 21 17:28:59.752252 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.752220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-557b5d69d4-jzgn8" event={"ID":"3f754a3b-88f2-4c0a-81b4-873ba820c568","Type":"ContainerStarted","Data":"fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c"}
Jul 21 17:28:59.752767 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.752265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-557b5d69d4-jzgn8" event={"ID":"3f754a3b-88f2-4c0a-81b4-873ba820c568","Type":"ContainerStarted","Data":"c3809a8555b4ec9908bbfbf78a2a66fe08a661079e649a1095a114f61e6d60e6"}
Jul 21 17:28:59.755216 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.755194    2579 generic.go:356] "Generic (PLEG): container finished" podID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerID="4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9" exitCode=0
Jul 21 17:28:59.755216 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.755212    2579 generic.go:356] "Generic (PLEG): container finished" podID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerID="b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b" exitCode=0
Jul 21 17:28:59.755216 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.755220    2579 generic.go:356] "Generic (PLEG): container finished" podID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerID="5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4" exitCode=0
Jul 21 17:28:59.755376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.755227    2579 generic.go:356] "Generic (PLEG): container finished" podID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerID="68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2" exitCode=0
Jul 21 17:28:59.755376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.755233    2579 generic.go:356] "Generic (PLEG): container finished" podID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerID="42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b" exitCode=0
Jul 21 17:28:59.755376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.755262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerDied","Data":"4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9"}
Jul 21 17:28:59.755376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.755321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerDied","Data":"b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b"}
Jul 21 17:28:59.755376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.755350    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerDied","Data":"5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4"}
Jul 21 17:28:59.755376 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.755376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerDied","Data":"68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2"}
Jul 21 17:28:59.755562 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.755403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerDied","Data":"42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b"}
Jul 21 17:28:59.773202 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:28:59.773158    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-557b5d69d4-jzgn8" podStartSLOduration=1.773145645 podStartE2EDuration="1.773145645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:28:59.772411231 +0000 UTC m=+165.331210248" watchObservedRunningTime="2026-07-21 17:28:59.773145645 +0000 UTC m=+165.331944628"
Jul 21 17:29:00.129518 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.129497    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:00.294448 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4l7l\" (UniqueName: \"kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-kube-api-access-z4l7l\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.294448 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294409    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-main-tls\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.294448 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294429    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-out\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.294448 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-web\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.294765 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294492    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.294765 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294530    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-trusted-ca-bundle\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.294765 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-tls-assets\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.294765 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294612    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.294765 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-main-db\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.294765 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294709    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-web-config\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.294765 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294740    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-cluster-tls-config\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.295097 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294783    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-volume\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.295097 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.294814    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-metrics-client-ca\") pod \"79fa2412-8f55-4919-8be1-3d703b6dd23c\" (UID: \"79fa2412-8f55-4919-8be1-3d703b6dd23c\") "
Jul 21 17:29:00.295471 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.295293    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:00.295471 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.295352    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:00.296150 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.296125    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:29:00.297700 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.297670    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-kube-api-access-z4l7l" (OuterVolumeSpecName: "kube-api-access-z4l7l") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "kube-api-access-z4l7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:29:00.297920 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.297728    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:00.297920 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.297763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:00.297920 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.297781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:00.297920 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.297787    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:29:00.297920 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.297790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-out" (OuterVolumeSpecName: "config-out") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:29:00.298584 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.298557    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-volume" (OuterVolumeSpecName: "config-volume") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:00.299166 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.299140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:00.301997 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.301900    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:00.307739 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.307713    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-web-config" (OuterVolumeSpecName: "web-config") pod "79fa2412-8f55-4919-8be1-3d703b6dd23c" (UID: "79fa2412-8f55-4919-8be1-3d703b6dd23c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:00.395634 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395615    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395634 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395633    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-tls-assets\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395642    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395671    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-alertmanager-main-db\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395682    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-web-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395690    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-cluster-tls-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395698    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-volume\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395706    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79fa2412-8f55-4919-8be1-3d703b6dd23c-metrics-client-ca\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395713    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4l7l\" (UniqueName: \"kubernetes.io/projected/79fa2412-8f55-4919-8be1-3d703b6dd23c-kube-api-access-z4l7l\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395721    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-main-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395728    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fa2412-8f55-4919-8be1-3d703b6dd23c-config-out\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395738    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.395774 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.395746    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79fa2412-8f55-4919-8be1-3d703b6dd23c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:29:00.762751 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.762721    2579 generic.go:356] "Generic (PLEG): container finished" podID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerID="14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678" exitCode=0
Jul 21 17:29:00.763188 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.762797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerDied","Data":"14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678"}
Jul 21 17:29:00.763188 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.762809    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:00.763188 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.762869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fa2412-8f55-4919-8be1-3d703b6dd23c","Type":"ContainerDied","Data":"8dad662f7c2fcd94a85876dafadc7263779541121c44f540c38882a257c1efd9"}
Jul 21 17:29:00.763188 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.762902    2579 scope.go:117] "RemoveContainer" containerID="4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9"
Jul 21 17:29:00.770952 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.770821    2579 scope.go:117] "RemoveContainer" containerID="b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b"
Jul 21 17:29:00.777985 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.777970    2579 scope.go:117] "RemoveContainer" containerID="5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4"
Jul 21 17:29:00.785102 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.785086    2579 scope.go:117] "RemoveContainer" containerID="14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678"
Jul 21 17:29:00.787522 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.787502    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:00.792024 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.792009    2579 scope.go:117] "RemoveContainer" containerID="68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2"
Jul 21 17:29:00.793972 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.793953    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:00.798331 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.798315    2579 scope.go:117] "RemoveContainer" containerID="42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b"
Jul 21 17:29:00.804698 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.804684    2579 scope.go:117] "RemoveContainer" containerID="c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee"
Jul 21 17:29:00.810913 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.810895    2579 scope.go:117] "RemoveContainer" containerID="4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9"
Jul 21 17:29:00.811150 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:29:00.811133    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9\": container with ID starting with 4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9 not found: ID does not exist" containerID="4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9"
Jul 21 17:29:00.811215 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.811161    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9"} err="failed to get container status \"4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9\": rpc error: code = NotFound desc = could not find container \"4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9\": container with ID starting with 4204fb760699f97cd094a6f1c68377f4a9043c403cd62736332cac8eb17d6ad9 not found: ID does not exist"
Jul 21 17:29:00.811215 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.811184    2579 scope.go:117] "RemoveContainer" containerID="b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b"
Jul 21 17:29:00.811411 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:29:00.811396    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b\": container with ID starting with b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b not found: ID does not exist" containerID="b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b"
Jul 21 17:29:00.811475 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.811419    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b"} err="failed to get container status \"b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b\": rpc error: code = NotFound desc = could not find container \"b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b\": container with ID starting with b5890587a4b67e0625aa9abf6f6a7ef0e90285048aa0f1cd860ad004fafade9b not found: ID does not exist"
Jul 21 17:29:00.811475 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.811440    2579 scope.go:117] "RemoveContainer" containerID="5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4"
Jul 21 17:29:00.811685 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:29:00.811664    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4\": container with ID starting with 5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4 not found: ID does not exist" containerID="5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4"
Jul 21 17:29:00.811727 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.811694    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4"} err="failed to get container status \"5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4\": rpc error: code = NotFound desc = could not find container \"5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4\": container with ID starting with 5ffe30617f8e0680e69093325467eb45c381b3c130152185eb15a5097dc7ccf4 not found: ID does not exist"
Jul 21 17:29:00.811727 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.811713    2579 scope.go:117] "RemoveContainer" containerID="14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678"
Jul 21 17:29:00.811942 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:29:00.811927    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678\": container with ID starting with 14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678 not found: ID does not exist" containerID="14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678"
Jul 21 17:29:00.811983 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.811945    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678"} err="failed to get container status \"14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678\": rpc error: code = NotFound desc = could not find container \"14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678\": container with ID starting with 14e6951faaf9648a420268e835336fe00371fe3939bd7cb29b87c05caac75678 not found: ID does not exist"
Jul 21 17:29:00.811983 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.811958    2579 scope.go:117] "RemoveContainer" containerID="68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2"
Jul 21 17:29:00.812172 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:29:00.812156    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2\": container with ID starting with 68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2 not found: ID does not exist" containerID="68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2"
Jul 21 17:29:00.812225 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.812182    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2"} err="failed to get container status \"68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2\": rpc error: code = NotFound desc = could not find container \"68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2\": container with ID starting with 68aaa4073c7fb1499035f2c4c5ebc7888c363a9217beb2ae86925ea72c6b50a2 not found: ID does not exist"
Jul 21 17:29:00.812225 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.812200    2579 scope.go:117] "RemoveContainer" containerID="42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b"
Jul 21 17:29:00.812412 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:29:00.812397    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b\": container with ID starting with 42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b not found: ID does not exist" containerID="42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b"
Jul 21 17:29:00.812449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.812417    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b"} err="failed to get container status \"42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b\": rpc error: code = NotFound desc = could not find container \"42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b\": container with ID starting with 42c0e536c5230c65d83ed1c492fb4b94d8f9399dc417d19648571bdddfd37d0b not found: ID does not exist"
Jul 21 17:29:00.812449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.812430    2579 scope.go:117] "RemoveContainer" containerID="c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee"
Jul 21 17:29:00.812688 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:29:00.812667    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee\": container with ID starting with c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee not found: ID does not exist" containerID="c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee"
Jul 21 17:29:00.812744 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.812694    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee"} err="failed to get container status \"c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee\": rpc error: code = NotFound desc = could not find container \"c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee\": container with ID starting with c72c570c0c086019a14ee2c77e5e2836f1904696fb72c3ab288058e34052f3ee not found: ID does not exist"
Jul 21 17:29:00.821805 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.821788    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:00.822044 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822018    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy-metric"
Jul 21 17:29:00.822044 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822029    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy-metric"
Jul 21 17:29:00.822044 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822042    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822047    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822060    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy-web"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822065    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy-web"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822072    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="init-config-reloader"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822077    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="init-config-reloader"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822083    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="alertmanager"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822088    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="alertmanager"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822092    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="config-reloader"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822097    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="config-reloader"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822102    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="prom-label-proxy"
Jul 21 17:29:00.822141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822106    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="prom-label-proxy"
Jul 21 17:29:00.822456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822153    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy-metric"
Jul 21 17:29:00.822456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822160    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="config-reloader"
Jul 21 17:29:00.822456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822166    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="prom-label-proxy"
Jul 21 17:29:00.822456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822171    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy"
Jul 21 17:29:00.822456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822176    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="kube-rbac-proxy-web"
Jul 21 17:29:00.822456 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.822181    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" containerName="alertmanager"
Jul 21 17:29:00.825630 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.825611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:00.828557 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.828538    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 21 17:29:00.828663 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.828540    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 21 17:29:00.828663 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.828607    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 21 17:29:00.828780 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.828761    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 21 17:29:00.828870 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.828851    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 21 17:29:00.828972 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.828939    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 21 17:29:00.828972 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.828951    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 21 17:29:00.829082 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.828970    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8nw54\""
Jul 21 17:29:00.829138 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.829123    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 21 17:29:00.836956 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.836931    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 21 17:29:00.842477 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:00.842456    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:01.000423 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000540 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000540 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000540 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185f77f4-1642-4f61-8dfc-b6d40ec472f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/185f77f4-1642-4f61-8dfc-b6d40ec472f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-web-config\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/185f77f4-1642-4f61-8dfc-b6d40ec472f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000850 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/185f77f4-1642-4f61-8dfc-b6d40ec472f3-config-out\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000850 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185f77f4-1642-4f61-8dfc-b6d40ec472f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000850 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v29rp\" (UniqueName: \"kubernetes.io/projected/185f77f4-1642-4f61-8dfc-b6d40ec472f3-kube-api-access-v29rp\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.000942 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.000865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.003302 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.003278    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79fa2412-8f55-4919-8be1-3d703b6dd23c" path="/var/lib/kubelet/pods/79fa2412-8f55-4919-8be1-3d703b6dd23c/volumes"
Jul 21 17:29:01.101953 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.101929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102060 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.101968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102060 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.101989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102060 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185f77f4-1642-4f61-8dfc-b6d40ec472f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102060 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/185f77f4-1642-4f61-8dfc-b6d40ec472f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102241 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102241 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-web-config\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102241 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/185f77f4-1642-4f61-8dfc-b6d40ec472f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102372 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102372 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/185f77f4-1642-4f61-8dfc-b6d40ec472f3-config-out\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102475 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185f77f4-1642-4f61-8dfc-b6d40ec472f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102578 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v29rp\" (UniqueName: \"kubernetes.io/projected/185f77f4-1642-4f61-8dfc-b6d40ec472f3-kube-api-access-v29rp\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.102778 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.103448 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.103420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/185f77f4-1642-4f61-8dfc-b6d40ec472f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.104716 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.102854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185f77f4-1642-4f61-8dfc-b6d40ec472f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.105093 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.104973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/185f77f4-1642-4f61-8dfc-b6d40ec472f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.105093 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.104992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.105093 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.104357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185f77f4-1642-4f61-8dfc-b6d40ec472f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.105413 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.105369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.105846 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.105801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.105948 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.105895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.106006 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.105948    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-web-config\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.106174 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.106155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.106612 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.106597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/185f77f4-1642-4f61-8dfc-b6d40ec472f3-config-out\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.107095 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.107075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/185f77f4-1642-4f61-8dfc-b6d40ec472f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.112767 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.112746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v29rp\" (UniqueName: \"kubernetes.io/projected/185f77f4-1642-4f61-8dfc-b6d40ec472f3-kube-api-access-v29rp\") pod \"alertmanager-main-0\" (UID: \"185f77f4-1642-4f61-8dfc-b6d40ec472f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.138487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.138469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:01.271202 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.271172    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:01.275968 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:29:01.275944    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185f77f4_1642_4f61_8dfc_b6d40ec472f3.slice/crio-350305d82f7f1a960ce7468033b1c589b785be9b10bd42007dca84f81e325d1e WatchSource:0}: Error finding container 350305d82f7f1a960ce7468033b1c589b785be9b10bd42007dca84f81e325d1e: Status 404 returned error can't find the container with id 350305d82f7f1a960ce7468033b1c589b785be9b10bd42007dca84f81e325d1e
Jul 21 17:29:01.768348 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.768313    2579 generic.go:356] "Generic (PLEG): container finished" podID="185f77f4-1642-4f61-8dfc-b6d40ec472f3" containerID="ff4a5fe314ec5273d6ae78dd9c620c62c4a64cb740f7c0f2c8d45e2fdfb54f06" exitCode=0
Jul 21 17:29:01.768793 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.768401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"185f77f4-1642-4f61-8dfc-b6d40ec472f3","Type":"ContainerDied","Data":"ff4a5fe314ec5273d6ae78dd9c620c62c4a64cb740f7c0f2c8d45e2fdfb54f06"}
Jul 21 17:29:01.768793 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:01.768475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"185f77f4-1642-4f61-8dfc-b6d40ec472f3","Type":"ContainerStarted","Data":"350305d82f7f1a960ce7468033b1c589b785be9b10bd42007dca84f81e325d1e"}
Jul 21 17:29:02.779953 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:02.779920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"185f77f4-1642-4f61-8dfc-b6d40ec472f3","Type":"ContainerStarted","Data":"46d4fdbf10288173b658ef76099954d72ed1af0619d0e2de051d34a4c356a88f"}
Jul 21 17:29:02.780345 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:02.780003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"185f77f4-1642-4f61-8dfc-b6d40ec472f3","Type":"ContainerStarted","Data":"5c0871ea16f8414c6d9362811f0a8c3f847c0f14532b65fa94cdf3b7d95a8c05"}
Jul 21 17:29:02.780345 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:02.780035    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"185f77f4-1642-4f61-8dfc-b6d40ec472f3","Type":"ContainerStarted","Data":"40efbf64be166b03daf5158327e088507e94be2b6593af42e5bc36a7c57ade17"}
Jul 21 17:29:02.780345 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:02.780059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"185f77f4-1642-4f61-8dfc-b6d40ec472f3","Type":"ContainerStarted","Data":"ad15ca55e5d6bf5a0be2a1ee607e5f236395b09c32f29a2cd53166f5650fa319"}
Jul 21 17:29:02.780345 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:02.780081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"185f77f4-1642-4f61-8dfc-b6d40ec472f3","Type":"ContainerStarted","Data":"20f3e79d24645cbc649e108f9cd5f3d440deddb980d2873566ce752f34a9e208"}
Jul 21 17:29:02.780345 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:02.780103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"185f77f4-1642-4f61-8dfc-b6d40ec472f3","Type":"ContainerStarted","Data":"943b06d020cfcee2eccc112e6c2220f6f1b2a165aea778c2a8e0e16e32ccae7a"}
Jul 21 17:29:02.810761 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:02.810705    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.7970803650000002 podStartE2EDuration="2.810691493s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.611128ms" isStatefulPod=false podCreationTimestamp="2026-07-21 17:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:29:02.810016455 +0000 UTC m=+168.368815439" watchObservedRunningTime="2026-07-21 17:29:02.810691493 +0000 UTC m=+168.369490475"
Jul 21 17:29:09.005781 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:09.005711    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:29:09.005781 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:09.005762    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:29:09.010139 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:09.010117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:29:09.812947 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:29:09.812917    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:30:26.307517 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.307482    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"]
Jul 21 17:30:26.310081 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.310064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.314581 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.314557    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fx69f\""
Jul 21 17:30:26.314581 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.314562    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 21 17:30:26.315544 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.315525    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 21 17:30:26.323459 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.323433    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"]
Jul 21 17:30:26.360201 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.360154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwpp2\" (UniqueName: \"kubernetes.io/projected/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-kube-api-access-fwpp2\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.360201 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.360202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.360409 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.360218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.461039 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.461007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpp2\" (UniqueName: \"kubernetes.io/projected/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-kube-api-access-fwpp2\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.461121 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.461052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.461121 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.461071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.461404 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.461389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.461440 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.461420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.470761 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.470743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpp2\" (UniqueName: \"kubernetes.io/projected/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-kube-api-access-fwpp2\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.619245 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.619209    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:26.756179 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:26.756142    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"]
Jul 21 17:30:26.760438 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:30:26.760401    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b35c077_dc9f_4e22_9f4f_ddd7640dcee7.slice/crio-2daf45a45de0859c83805bc9da4859fccaa6521453897000b69417422a67fbb4 WatchSource:0}: Error finding container 2daf45a45de0859c83805bc9da4859fccaa6521453897000b69417422a67fbb4: Status 404 returned error can't find the container with id 2daf45a45de0859c83805bc9da4859fccaa6521453897000b69417422a67fbb4
Jul 21 17:30:27.085198 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:27.085111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l" event={"ID":"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7","Type":"ContainerStarted","Data":"2daf45a45de0859c83805bc9da4859fccaa6521453897000b69417422a67fbb4"}
Jul 21 17:30:34.117095 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:34.117056    2579 generic.go:356] "Generic (PLEG): container finished" podID="9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" containerID="75deea97c83d366b4a790703535896aabb1e4083627e8fbe021a078535fead7d" exitCode=0
Jul 21 17:30:34.117483 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:34.117141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l" event={"ID":"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7","Type":"ContainerDied","Data":"75deea97c83d366b4a790703535896aabb1e4083627e8fbe021a078535fead7d"}
Jul 21 17:30:38.135546 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:38.135509    2579 generic.go:356] "Generic (PLEG): container finished" podID="9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" containerID="cd7fe61b68cde92a7cacbe008ee58afd688e61df70d4082e5f39ad481734da6d" exitCode=0
Jul 21 17:30:38.135959 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:38.135549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l" event={"ID":"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7","Type":"ContainerDied","Data":"cd7fe61b68cde92a7cacbe008ee58afd688e61df70d4082e5f39ad481734da6d"}
Jul 21 17:30:45.167362 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:45.167326    2579 generic.go:356] "Generic (PLEG): container finished" podID="9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" containerID="dfb1585ce7ba305233ddc2f485a5d613665e8b953feffd7e0a924f74bd407966" exitCode=0
Jul 21 17:30:45.167754 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:45.167382    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l" event={"ID":"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7","Type":"ContainerDied","Data":"dfb1585ce7ba305233ddc2f485a5d613665e8b953feffd7e0a924f74bd407966"}
Jul 21 17:30:46.310103 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:46.310076    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:46.431284 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:46.431189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-bundle\") pod \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") "
Jul 21 17:30:46.431284 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:46.431242    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwpp2\" (UniqueName: \"kubernetes.io/projected/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-kube-api-access-fwpp2\") pod \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") "
Jul 21 17:30:46.431284 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:46.431259    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-util\") pod \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\" (UID: \"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7\") "
Jul 21 17:30:46.431893 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:46.431867    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-bundle" (OuterVolumeSpecName: "bundle") pod "9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" (UID: "9b35c077-dc9f-4e22-9f4f-ddd7640dcee7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:30:46.433406 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:46.433385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-kube-api-access-fwpp2" (OuterVolumeSpecName: "kube-api-access-fwpp2") pod "9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" (UID: "9b35c077-dc9f-4e22-9f4f-ddd7640dcee7"). InnerVolumeSpecName "kube-api-access-fwpp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:30:46.436162 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:46.436133    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-util" (OuterVolumeSpecName: "util") pod "9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" (UID: "9b35c077-dc9f-4e22-9f4f-ddd7640dcee7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:30:46.532348 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:46.532310    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-bundle\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:30:46.532348 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:46.532340    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwpp2\" (UniqueName: \"kubernetes.io/projected/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-kube-api-access-fwpp2\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:30:46.532348 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:46.532350    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b35c077-dc9f-4e22-9f4f-ddd7640dcee7-util\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:30:47.176335 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:47.176302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l" event={"ID":"9b35c077-dc9f-4e22-9f4f-ddd7640dcee7","Type":"ContainerDied","Data":"2daf45a45de0859c83805bc9da4859fccaa6521453897000b69417422a67fbb4"}
Jul 21 17:30:47.176335 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:47.176341    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vgl2l"
Jul 21 17:30:47.176534 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:30:47.176355    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2daf45a45de0859c83805bc9da4859fccaa6521453897000b69417422a67fbb4"
Jul 21 17:31:00.149475 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.149440    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"]
Jul 21 17:31:00.149954 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.149735    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" containerName="pull"
Jul 21 17:31:00.149954 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.149746    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" containerName="pull"
Jul 21 17:31:00.149954 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.149755    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" containerName="extract"
Jul 21 17:31:00.149954 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.149760    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" containerName="extract"
Jul 21 17:31:00.149954 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.149773    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" containerName="util"
Jul 21 17:31:00.149954 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.149778    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" containerName="util"
Jul 21 17:31:00.149954 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.149823    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b35c077-dc9f-4e22-9f4f-ddd7640dcee7" containerName="extract"
Jul 21 17:31:00.153905 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.153890    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.158831 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.158813    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 21 17:31:00.158934 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.158866    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 21 17:31:00.159809 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.159791    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 21 17:31:00.159809 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.159804    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-4czn5\""
Jul 21 17:31:00.160431 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.160411    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 21 17:31:00.160542 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.160513    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 21 17:31:00.170732 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.170710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"]
Jul 21 17:31:00.237894 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.237867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/62cb6d51-cf33-4854-9990-4a26bdf10df6-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tcmlc\" (UID: \"62cb6d51-cf33-4854-9990-4a26bdf10df6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.238054 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.237906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/62cb6d51-cf33-4854-9990-4a26bdf10df6-certificates\") pod \"keda-metrics-apiserver-55557df8c-tcmlc\" (UID: \"62cb6d51-cf33-4854-9990-4a26bdf10df6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.238054 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.237942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mshd\" (UniqueName: \"kubernetes.io/projected/62cb6d51-cf33-4854-9990-4a26bdf10df6-kube-api-access-9mshd\") pod \"keda-metrics-apiserver-55557df8c-tcmlc\" (UID: \"62cb6d51-cf33-4854-9990-4a26bdf10df6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.323859 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.323828    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-95pjf"]
Jul 21 17:31:00.326015 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.325998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-95pjf"
Jul 21 17:31:00.329637 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.329608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 21 17:31:00.338426 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.338392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/62cb6d51-cf33-4854-9990-4a26bdf10df6-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tcmlc\" (UID: \"62cb6d51-cf33-4854-9990-4a26bdf10df6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.338586 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.338456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/62cb6d51-cf33-4854-9990-4a26bdf10df6-certificates\") pod \"keda-metrics-apiserver-55557df8c-tcmlc\" (UID: \"62cb6d51-cf33-4854-9990-4a26bdf10df6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.338586 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.338511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mshd\" (UniqueName: \"kubernetes.io/projected/62cb6d51-cf33-4854-9990-4a26bdf10df6-kube-api-access-9mshd\") pod \"keda-metrics-apiserver-55557df8c-tcmlc\" (UID: \"62cb6d51-cf33-4854-9990-4a26bdf10df6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.338860 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.338839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/62cb6d51-cf33-4854-9990-4a26bdf10df6-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tcmlc\" (UID: \"62cb6d51-cf33-4854-9990-4a26bdf10df6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.341780 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.341757    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-95pjf"]
Jul 21 17:31:00.344203 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.344181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/62cb6d51-cf33-4854-9990-4a26bdf10df6-certificates\") pod \"keda-metrics-apiserver-55557df8c-tcmlc\" (UID: \"62cb6d51-cf33-4854-9990-4a26bdf10df6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.351345 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.351322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mshd\" (UniqueName: \"kubernetes.io/projected/62cb6d51-cf33-4854-9990-4a26bdf10df6-kube-api-access-9mshd\") pod \"keda-metrics-apiserver-55557df8c-tcmlc\" (UID: \"62cb6d51-cf33-4854-9990-4a26bdf10df6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.439132 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.439034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvpr\" (UniqueName: \"kubernetes.io/projected/817b04b5-84f7-47f2-b646-265600887819-kube-api-access-kvvpr\") pod \"keda-admission-56f9798bb4-95pjf\" (UID: \"817b04b5-84f7-47f2-b646-265600887819\") " pod="openshift-keda/keda-admission-56f9798bb4-95pjf"
Jul 21 17:31:00.439132 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.439086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/817b04b5-84f7-47f2-b646-265600887819-certificates\") pod \"keda-admission-56f9798bb4-95pjf\" (UID: \"817b04b5-84f7-47f2-b646-265600887819\") " pod="openshift-keda/keda-admission-56f9798bb4-95pjf"
Jul 21 17:31:00.464021 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.463987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:00.539723 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.539693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvpr\" (UniqueName: \"kubernetes.io/projected/817b04b5-84f7-47f2-b646-265600887819-kube-api-access-kvvpr\") pod \"keda-admission-56f9798bb4-95pjf\" (UID: \"817b04b5-84f7-47f2-b646-265600887819\") " pod="openshift-keda/keda-admission-56f9798bb4-95pjf"
Jul 21 17:31:00.539888 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.539758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/817b04b5-84f7-47f2-b646-265600887819-certificates\") pod \"keda-admission-56f9798bb4-95pjf\" (UID: \"817b04b5-84f7-47f2-b646-265600887819\") " pod="openshift-keda/keda-admission-56f9798bb4-95pjf"
Jul 21 17:31:00.542517 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.542493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/817b04b5-84f7-47f2-b646-265600887819-certificates\") pod \"keda-admission-56f9798bb4-95pjf\" (UID: \"817b04b5-84f7-47f2-b646-265600887819\") " pod="openshift-keda/keda-admission-56f9798bb4-95pjf"
Jul 21 17:31:00.550302 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.550260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvpr\" (UniqueName: \"kubernetes.io/projected/817b04b5-84f7-47f2-b646-265600887819-kube-api-access-kvvpr\") pod \"keda-admission-56f9798bb4-95pjf\" (UID: \"817b04b5-84f7-47f2-b646-265600887819\") " pod="openshift-keda/keda-admission-56f9798bb4-95pjf"
Jul 21 17:31:00.617346 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.617319    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"]
Jul 21 17:31:00.620637 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:31:00.620610    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cb6d51_cf33_4854_9990_4a26bdf10df6.slice/crio-3e8d7da20dcff3dee2952ba7f0464f6e0a38a8e3ba4ae822d4640f688c17224c WatchSource:0}: Error finding container 3e8d7da20dcff3dee2952ba7f0464f6e0a38a8e3ba4ae822d4640f688c17224c: Status 404 returned error can't find the container with id 3e8d7da20dcff3dee2952ba7f0464f6e0a38a8e3ba4ae822d4640f688c17224c
Jul 21 17:31:00.637993 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.637972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-95pjf"
Jul 21 17:31:00.783277 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:00.783235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-95pjf"]
Jul 21 17:31:00.787499 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:31:00.787468    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817b04b5_84f7_47f2_b646_265600887819.slice/crio-311df9b51c112d8f8c88f8c0064438e534b2e4ea3429fffc2cf91d557b7c4687 WatchSource:0}: Error finding container 311df9b51c112d8f8c88f8c0064438e534b2e4ea3429fffc2cf91d557b7c4687: Status 404 returned error can't find the container with id 311df9b51c112d8f8c88f8c0064438e534b2e4ea3429fffc2cf91d557b7c4687
Jul 21 17:31:01.229610 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:01.229572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc" event={"ID":"62cb6d51-cf33-4854-9990-4a26bdf10df6","Type":"ContainerStarted","Data":"3e8d7da20dcff3dee2952ba7f0464f6e0a38a8e3ba4ae822d4640f688c17224c"}
Jul 21 17:31:01.230763 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:01.230735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-95pjf" event={"ID":"817b04b5-84f7-47f2-b646-265600887819","Type":"ContainerStarted","Data":"311df9b51c112d8f8c88f8c0064438e534b2e4ea3429fffc2cf91d557b7c4687"}
Jul 21 17:31:04.245494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:04.245457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc" event={"ID":"62cb6d51-cf33-4854-9990-4a26bdf10df6","Type":"ContainerStarted","Data":"009ba64169a2abcfbfd47d8760cee4724234a70ccdb139a1672724b6c98f6e17"}
Jul 21 17:31:04.245966 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:04.245521    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:04.246843 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:04.246819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-95pjf" event={"ID":"817b04b5-84f7-47f2-b646-265600887819","Type":"ContainerStarted","Data":"e08420b6b8c0176bde9edc97e130d97c9928e372ef1f2309d043a2ca6a2ec59f"}
Jul 21 17:31:04.246972 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:04.246870    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-95pjf"
Jul 21 17:31:04.264723 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:04.264678    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc" podStartSLOduration=1.269805716 podStartE2EDuration="4.264663343s" totalImagesPullingTime="2.994857627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:31:04.263260743 +0000 UTC m=+289.822059728" watchObservedRunningTime="2026-07-21 17:31:04.264663343 +0000 UTC m=+289.823462318"
Jul 21 17:31:04.281299 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:04.281262    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-95pjf" podStartSLOduration=1.455460797 podStartE2EDuration="4.281251395s" totalImagesPullingTime="2.825790598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:31:04.280711325 +0000 UTC m=+289.839510309" watchObservedRunningTime="2026-07-21 17:31:04.281251395 +0000 UTC m=+289.840050378"
Jul 21 17:31:14.844534 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:14.844506    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 21 17:31:15.255763 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:15.255737    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tcmlc"
Jul 21 17:31:25.254265 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:31:25.254234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-95pjf"
Jul 21 17:32:16.729524 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.729484    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6b69bf598b-grmvz"]
Jul 21 17:32:16.731872 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.731851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:32:16.734861 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.734839    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 21 17:32:16.735077 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.735064    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 21 17:32:16.736314 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.736296    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 21 17:32:16.736366 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.736327    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-xtpb9\""
Jul 21 17:32:16.744831 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.744809    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69bf598b-grmvz"]
Jul 21 17:32:16.767635 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.767615    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-48vr9"]
Jul 21 17:32:16.769737 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.769722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-48vr9"
Jul 21 17:32:16.772807 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.772792    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 21 17:32:16.772892 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.772819    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ckl7w\""
Jul 21 17:32:16.783998 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.783975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-48vr9"]
Jul 21 17:32:16.818934 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.818911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-cert\") pod \"kserve-controller-manager-6b69bf598b-grmvz\" (UID: \"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7\") " pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:32:16.819049 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.818972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdn9c\" (UniqueName: \"kubernetes.io/projected/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-kube-api-access-jdn9c\") pod \"kserve-controller-manager-6b69bf598b-grmvz\" (UID: \"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7\") " pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:32:16.919671 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.919626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdn9c\" (UniqueName: \"kubernetes.io/projected/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-kube-api-access-jdn9c\") pod \"kserve-controller-manager-6b69bf598b-grmvz\" (UID: \"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7\") " pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:32:16.919809 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.919684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d29aa7fb-c251-4e97-bad0-0a72d147d211-data\") pod \"seaweedfs-64568bcd49-48vr9\" (UID: \"d29aa7fb-c251-4e97-bad0-0a72d147d211\") " pod="kserve/seaweedfs-64568bcd49-48vr9"
Jul 21 17:32:16.919809 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.919717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-cert\") pod \"kserve-controller-manager-6b69bf598b-grmvz\" (UID: \"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7\") " pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:32:16.919809 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.919749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skkwl\" (UniqueName: \"kubernetes.io/projected/d29aa7fb-c251-4e97-bad0-0a72d147d211-kube-api-access-skkwl\") pod \"seaweedfs-64568bcd49-48vr9\" (UID: \"d29aa7fb-c251-4e97-bad0-0a72d147d211\") " pod="kserve/seaweedfs-64568bcd49-48vr9"
Jul 21 17:32:16.922052 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.922032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-cert\") pod \"kserve-controller-manager-6b69bf598b-grmvz\" (UID: \"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7\") " pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:32:16.927995 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:16.927970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdn9c\" (UniqueName: \"kubernetes.io/projected/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-kube-api-access-jdn9c\") pod \"kserve-controller-manager-6b69bf598b-grmvz\" (UID: \"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7\") " pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:32:17.020664 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.020585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d29aa7fb-c251-4e97-bad0-0a72d147d211-data\") pod \"seaweedfs-64568bcd49-48vr9\" (UID: \"d29aa7fb-c251-4e97-bad0-0a72d147d211\") " pod="kserve/seaweedfs-64568bcd49-48vr9"
Jul 21 17:32:17.020664 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.020645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skkwl\" (UniqueName: \"kubernetes.io/projected/d29aa7fb-c251-4e97-bad0-0a72d147d211-kube-api-access-skkwl\") pod \"seaweedfs-64568bcd49-48vr9\" (UID: \"d29aa7fb-c251-4e97-bad0-0a72d147d211\") " pod="kserve/seaweedfs-64568bcd49-48vr9"
Jul 21 17:32:17.020982 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.020962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d29aa7fb-c251-4e97-bad0-0a72d147d211-data\") pod \"seaweedfs-64568bcd49-48vr9\" (UID: \"d29aa7fb-c251-4e97-bad0-0a72d147d211\") " pod="kserve/seaweedfs-64568bcd49-48vr9"
Jul 21 17:32:17.030605 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.030583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skkwl\" (UniqueName: \"kubernetes.io/projected/d29aa7fb-c251-4e97-bad0-0a72d147d211-kube-api-access-skkwl\") pod \"seaweedfs-64568bcd49-48vr9\" (UID: \"d29aa7fb-c251-4e97-bad0-0a72d147d211\") " pod="kserve/seaweedfs-64568bcd49-48vr9"
Jul 21 17:32:17.041402 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.041382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:32:17.081752 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.081728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-48vr9"
Jul 21 17:32:17.181426 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.181349    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69bf598b-grmvz"]
Jul 21 17:32:17.183951 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:32:17.183926    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cec9aa4_08ac_48cb_8893_fe7eedaf16f7.slice/crio-3da14eac2351ca263c7afe00c7fd3bb34cf459d956171e37df0e39755ad456d7 WatchSource:0}: Error finding container 3da14eac2351ca263c7afe00c7fd3bb34cf459d956171e37df0e39755ad456d7: Status 404 returned error can't find the container with id 3da14eac2351ca263c7afe00c7fd3bb34cf459d956171e37df0e39755ad456d7
Jul 21 17:32:17.185501 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.185483    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:32:17.236560 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.236539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-48vr9"]
Jul 21 17:32:17.238497 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:32:17.238471    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29aa7fb_c251_4e97_bad0_0a72d147d211.slice/crio-b41f455d978e44b63ce31122fd714a7b79952088b1b344004a285ee25c0ab246 WatchSource:0}: Error finding container b41f455d978e44b63ce31122fd714a7b79952088b1b344004a285ee25c0ab246: Status 404 returned error can't find the container with id b41f455d978e44b63ce31122fd714a7b79952088b1b344004a285ee25c0ab246
Jul 21 17:32:17.535078 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.535047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-48vr9" event={"ID":"d29aa7fb-c251-4e97-bad0-0a72d147d211","Type":"ContainerStarted","Data":"b41f455d978e44b63ce31122fd714a7b79952088b1b344004a285ee25c0ab246"}
Jul 21 17:32:17.536144 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:17.536113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz" event={"ID":"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7","Type":"ContainerStarted","Data":"3da14eac2351ca263c7afe00c7fd3bb34cf459d956171e37df0e39755ad456d7"}
Jul 21 17:32:21.558426 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:21.558375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz" event={"ID":"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7","Type":"ContainerStarted","Data":"2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d"}
Jul 21 17:32:21.558895 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:21.558456    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:32:21.559897 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:21.559873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-48vr9" event={"ID":"d29aa7fb-c251-4e97-bad0-0a72d147d211","Type":"ContainerStarted","Data":"0ae79fc9c82f58794f31f7f0a9568504e9d67293bf34592f4bc68496f1e1a366"}
Jul 21 17:32:21.559996 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:21.559923    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-48vr9"
Jul 21 17:32:21.577276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:21.577237    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz" podStartSLOduration=2.222371651 podStartE2EDuration="5.577217058s" totalImagesPullingTime="3.354845407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:32:21.576347679 +0000 UTC m=+367.135146663" watchObservedRunningTime="2026-07-21 17:32:21.577217058 +0000 UTC m=+367.136016042"
Jul 21 17:32:21.593705 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:21.593640    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-48vr9" podStartSLOduration=2.224878015 podStartE2EDuration="5.59362743s" totalImagesPullingTime="3.368749415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:32:21.592724985 +0000 UTC m=+367.151523969" watchObservedRunningTime="2026-07-21 17:32:21.59362743 +0000 UTC m=+367.152426414"
Jul 21 17:32:27.566538 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:27.566511    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-48vr9"
Jul 21 17:32:52.568977 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:52.568948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:32:54.081398 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.081369    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b454c7bc8-gqqgd"]
Jul 21 17:32:54.086543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.086520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.103400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.103376    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b454c7bc8-gqqgd"]
Jul 21 17:32:54.107092 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.107070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-oauth-serving-cert\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.107186 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.107108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-console-config\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.107186 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.107125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdtkr\" (UniqueName: \"kubernetes.io/projected/2b37ae05-94f4-4157-a8b5-c612186acd24-kube-api-access-gdtkr\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.107255 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.107209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b37ae05-94f4-4157-a8b5-c612186acd24-console-oauth-config\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.107255 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.107249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-trusted-ca-bundle\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.107318 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.107279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-service-ca\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.107318 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.107309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b37ae05-94f4-4157-a8b5-c612186acd24-console-serving-cert\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.208350 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.208320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-oauth-serving-cert\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.208520 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.208363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-console-config\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.208520 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.208382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdtkr\" (UniqueName: \"kubernetes.io/projected/2b37ae05-94f4-4157-a8b5-c612186acd24-kube-api-access-gdtkr\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.208520 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.208407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b37ae05-94f4-4157-a8b5-c612186acd24-console-oauth-config\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.208520 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.208427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-trusted-ca-bundle\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.208520 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.208447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-service-ca\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.208520 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.208468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b37ae05-94f4-4157-a8b5-c612186acd24-console-serving-cert\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.209177 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.209136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-console-config\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.209289 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.209242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-oauth-serving-cert\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.209289 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.209269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-trusted-ca-bundle\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.209360 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.209300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b37ae05-94f4-4157-a8b5-c612186acd24-service-ca\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.210830 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.210806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b37ae05-94f4-4157-a8b5-c612186acd24-console-serving-cert\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.210915 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.210876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b37ae05-94f4-4157-a8b5-c612186acd24-console-oauth-config\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.218871 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.218850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdtkr\" (UniqueName: \"kubernetes.io/projected/2b37ae05-94f4-4157-a8b5-c612186acd24-kube-api-access-gdtkr\") pod \"console-6b454c7bc8-gqqgd\" (UID: \"2b37ae05-94f4-4157-a8b5-c612186acd24\") " pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.396861 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.396838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:32:54.528114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.528092    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b454c7bc8-gqqgd"]
Jul 21 17:32:54.530085 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:32:54.530058    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b37ae05_94f4_4157_a8b5_c612186acd24.slice/crio-cc627f17c1c4a114b048247ebe33304df487b23f188fc876a839c725641a7d01 WatchSource:0}: Error finding container cc627f17c1c4a114b048247ebe33304df487b23f188fc876a839c725641a7d01: Status 404 returned error can't find the container with id cc627f17c1c4a114b048247ebe33304df487b23f188fc876a839c725641a7d01
Jul 21 17:32:54.689227 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.689136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b454c7bc8-gqqgd" event={"ID":"2b37ae05-94f4-4157-a8b5-c612186acd24","Type":"ContainerStarted","Data":"0b3dc2a68b88c8cf864fe006a404e198f37bd02ecfd5e1c9bbbdc09e26529af9"}
Jul 21 17:32:54.689227 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.689184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b454c7bc8-gqqgd" event={"ID":"2b37ae05-94f4-4157-a8b5-c612186acd24","Type":"ContainerStarted","Data":"cc627f17c1c4a114b048247ebe33304df487b23f188fc876a839c725641a7d01"}
Jul 21 17:32:54.714622 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:54.714578    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b454c7bc8-gqqgd" podStartSLOduration=0.71456323 podStartE2EDuration="714.56323ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:32:54.713082793 +0000 UTC m=+400.271881771" watchObservedRunningTime="2026-07-21 17:32:54.71456323 +0000 UTC m=+400.273362213"
Jul 21 17:32:59.908463 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:59.908428    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6b69bf598b-grmvz"]
Jul 21 17:32:59.908948 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:59.908669    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz" podUID="8cec9aa4-08ac-48cb-8893-fe7eedaf16f7" containerName="manager" containerID="cri-o://2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d" gracePeriod=10
Jul 21 17:32:59.932416 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:59.932382    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6b69bf598b-f697k"]
Jul 21 17:32:59.934863 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:59.934847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69bf598b-f697k"
Jul 21 17:32:59.945263 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:59.945233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69bf598b-f697k"]
Jul 21 17:32:59.948176 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:59.948158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stx5s\" (UniqueName: \"kubernetes.io/projected/41c976e9-cec7-4777-a011-0b2987f4fc2a-kube-api-access-stx5s\") pod \"kserve-controller-manager-6b69bf598b-f697k\" (UID: \"41c976e9-cec7-4777-a011-0b2987f4fc2a\") " pod="kserve/kserve-controller-manager-6b69bf598b-f697k"
Jul 21 17:32:59.948271 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:32:59.948212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41c976e9-cec7-4777-a011-0b2987f4fc2a-cert\") pod \"kserve-controller-manager-6b69bf598b-f697k\" (UID: \"41c976e9-cec7-4777-a011-0b2987f4fc2a\") " pod="kserve/kserve-controller-manager-6b69bf598b-f697k"
Jul 21 17:33:00.048611 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.048572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41c976e9-cec7-4777-a011-0b2987f4fc2a-cert\") pod \"kserve-controller-manager-6b69bf598b-f697k\" (UID: \"41c976e9-cec7-4777-a011-0b2987f4fc2a\") " pod="kserve/kserve-controller-manager-6b69bf598b-f697k"
Jul 21 17:33:00.048809 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.048645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stx5s\" (UniqueName: \"kubernetes.io/projected/41c976e9-cec7-4777-a011-0b2987f4fc2a-kube-api-access-stx5s\") pod \"kserve-controller-manager-6b69bf598b-f697k\" (UID: \"41c976e9-cec7-4777-a011-0b2987f4fc2a\") " pod="kserve/kserve-controller-manager-6b69bf598b-f697k"
Jul 21 17:33:00.050825 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.050801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41c976e9-cec7-4777-a011-0b2987f4fc2a-cert\") pod \"kserve-controller-manager-6b69bf598b-f697k\" (UID: \"41c976e9-cec7-4777-a011-0b2987f4fc2a\") " pod="kserve/kserve-controller-manager-6b69bf598b-f697k"
Jul 21 17:33:00.057322 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.057289    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stx5s\" (UniqueName: \"kubernetes.io/projected/41c976e9-cec7-4777-a011-0b2987f4fc2a-kube-api-access-stx5s\") pod \"kserve-controller-manager-6b69bf598b-f697k\" (UID: \"41c976e9-cec7-4777-a011-0b2987f4fc2a\") " pod="kserve/kserve-controller-manager-6b69bf598b-f697k"
Jul 21 17:33:00.143047 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.143027    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:33:00.249912 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.249838    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdn9c\" (UniqueName: \"kubernetes.io/projected/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-kube-api-access-jdn9c\") pod \"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7\" (UID: \"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7\") "
Jul 21 17:33:00.249912 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.249884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-cert\") pod \"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7\" (UID: \"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7\") "
Jul 21 17:33:00.251941 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.251913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-cert" (OuterVolumeSpecName: "cert") pod "8cec9aa4-08ac-48cb-8893-fe7eedaf16f7" (UID: "8cec9aa4-08ac-48cb-8893-fe7eedaf16f7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:33:00.252055 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.251981    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-kube-api-access-jdn9c" (OuterVolumeSpecName: "kube-api-access-jdn9c") pod "8cec9aa4-08ac-48cb-8893-fe7eedaf16f7" (UID: "8cec9aa4-08ac-48cb-8893-fe7eedaf16f7"). InnerVolumeSpecName "kube-api-access-jdn9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:33:00.293287 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.293264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69bf598b-f697k"
Jul 21 17:33:00.350715 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.350636    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:33:00.350715 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.350687    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdn9c\" (UniqueName: \"kubernetes.io/projected/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7-kube-api-access-jdn9c\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:33:00.417160 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.417136    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69bf598b-f697k"]
Jul 21 17:33:00.419106 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:33:00.419080    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c976e9_cec7_4777_a011_0b2987f4fc2a.slice/crio-0f1ec5980d096b8eb8e68b506d43d4f4d368c63133e1a37a676ff5fce2411429 WatchSource:0}: Error finding container 0f1ec5980d096b8eb8e68b506d43d4f4d368c63133e1a37a676ff5fce2411429: Status 404 returned error can't find the container with id 0f1ec5980d096b8eb8e68b506d43d4f4d368c63133e1a37a676ff5fce2411429
Jul 21 17:33:00.715257 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.715227    2579 generic.go:356] "Generic (PLEG): container finished" podID="8cec9aa4-08ac-48cb-8893-fe7eedaf16f7" containerID="2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d" exitCode=0
Jul 21 17:33:00.715415 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.715273    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz"
Jul 21 17:33:00.715415 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.715313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz" event={"ID":"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7","Type":"ContainerDied","Data":"2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d"}
Jul 21 17:33:00.715415 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.715366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69bf598b-grmvz" event={"ID":"8cec9aa4-08ac-48cb-8893-fe7eedaf16f7","Type":"ContainerDied","Data":"3da14eac2351ca263c7afe00c7fd3bb34cf459d956171e37df0e39755ad456d7"}
Jul 21 17:33:00.715415 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.715395    2579 scope.go:117] "RemoveContainer" containerID="2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d"
Jul 21 17:33:00.716571 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.716551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69bf598b-f697k" event={"ID":"41c976e9-cec7-4777-a011-0b2987f4fc2a","Type":"ContainerStarted","Data":"0f1ec5980d096b8eb8e68b506d43d4f4d368c63133e1a37a676ff5fce2411429"}
Jul 21 17:33:00.723689 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.723672    2579 scope.go:117] "RemoveContainer" containerID="2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d"
Jul 21 17:33:00.723943 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:33:00.723927    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d\": container with ID starting with 2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d not found: ID does not exist" containerID="2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d"
Jul 21 17:33:00.723981 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.723953    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d"} err="failed to get container status \"2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d\": rpc error: code = NotFound desc = could not find container \"2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d\": container with ID starting with 2062b10d96a028735bb3f1407b4cea7af518e643c316d21d338a2db37fc71e8d not found: ID does not exist"
Jul 21 17:33:00.738110 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.738090    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6b69bf598b-grmvz"]
Jul 21 17:33:00.741888 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:00.741870    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6b69bf598b-grmvz"]
Jul 21 17:33:01.003498 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:01.003428    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cec9aa4-08ac-48cb-8893-fe7eedaf16f7" path="/var/lib/kubelet/pods/8cec9aa4-08ac-48cb-8893-fe7eedaf16f7/volumes"
Jul 21 17:33:01.723142 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:01.723110    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69bf598b-f697k" event={"ID":"41c976e9-cec7-4777-a011-0b2987f4fc2a","Type":"ContainerStarted","Data":"cef954648a38dc0af1a53ff2a2d536a1895f10bf2030284a865479b9c3bd692c"}
Jul 21 17:33:01.723298 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:01.723236    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6b69bf598b-f697k"
Jul 21 17:33:01.741345 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:01.741305    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6b69bf598b-f697k" podStartSLOduration=2.374641382 podStartE2EDuration="2.741291677s" totalImagesPullingTime="366.650295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:33:01.739538375 +0000 UTC m=+407.298337359" watchObservedRunningTime="2026-07-21 17:33:01.741291677 +0000 UTC m=+407.300090702"
Jul 21 17:33:04.397705 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:04.397670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:33:04.398068 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:04.397753    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:33:04.402425 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:04.402405    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:33:04.739496 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:04.739410    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b454c7bc8-gqqgd"
Jul 21 17:33:04.790204 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:04.790176    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-557b5d69d4-jzgn8"]
Jul 21 17:33:29.813029 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:29.812986    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-557b5d69d4-jzgn8" podUID="3f754a3b-88f2-4c0a-81b4-873ba820c568" containerName="console" containerID="cri-o://fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c" gracePeriod=15
Jul 21 17:33:30.054785 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.054759    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:33:30.188935 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.188906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-config\") pod \"3f754a3b-88f2-4c0a-81b4-873ba820c568\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") "
Jul 21 17:33:30.189087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.188964    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-serving-cert\") pod \"3f754a3b-88f2-4c0a-81b4-873ba820c568\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") "
Jul 21 17:33:30.189087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.188987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-oauth-config\") pod \"3f754a3b-88f2-4c0a-81b4-873ba820c568\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") "
Jul 21 17:33:30.189087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.189002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-trusted-ca-bundle\") pod \"3f754a3b-88f2-4c0a-81b4-873ba820c568\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") "
Jul 21 17:33:30.189087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.189034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st4pk\" (UniqueName: \"kubernetes.io/projected/3f754a3b-88f2-4c0a-81b4-873ba820c568-kube-api-access-st4pk\") pod \"3f754a3b-88f2-4c0a-81b4-873ba820c568\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") "
Jul 21 17:33:30.189087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.189050    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-oauth-serving-cert\") pod \"3f754a3b-88f2-4c0a-81b4-873ba820c568\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") "
Jul 21 17:33:30.189087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.189075    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-service-ca\") pod \"3f754a3b-88f2-4c0a-81b4-873ba820c568\" (UID: \"3f754a3b-88f2-4c0a-81b4-873ba820c568\") "
Jul 21 17:33:30.189464 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.189430    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-config" (OuterVolumeSpecName: "console-config") pod "3f754a3b-88f2-4c0a-81b4-873ba820c568" (UID: "3f754a3b-88f2-4c0a-81b4-873ba820c568"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:33:30.189550 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.189521    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3f754a3b-88f2-4c0a-81b4-873ba820c568" (UID: "3f754a3b-88f2-4c0a-81b4-873ba820c568"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:33:30.189613 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.189552    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-service-ca" (OuterVolumeSpecName: "service-ca") pod "3f754a3b-88f2-4c0a-81b4-873ba820c568" (UID: "3f754a3b-88f2-4c0a-81b4-873ba820c568"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:33:30.189613 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.189548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3f754a3b-88f2-4c0a-81b4-873ba820c568" (UID: "3f754a3b-88f2-4c0a-81b4-873ba820c568"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:33:30.191159 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.191139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f754a3b-88f2-4c0a-81b4-873ba820c568-kube-api-access-st4pk" (OuterVolumeSpecName: "kube-api-access-st4pk") pod "3f754a3b-88f2-4c0a-81b4-873ba820c568" (UID: "3f754a3b-88f2-4c0a-81b4-873ba820c568"). InnerVolumeSpecName "kube-api-access-st4pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:33:30.191256 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.191172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3f754a3b-88f2-4c0a-81b4-873ba820c568" (UID: "3f754a3b-88f2-4c0a-81b4-873ba820c568"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:33:30.191256 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.191234    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3f754a3b-88f2-4c0a-81b4-873ba820c568" (UID: "3f754a3b-88f2-4c0a-81b4-873ba820c568"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:33:30.289893 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.289859    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-oauth-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:33:30.289893 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.289890    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-trusted-ca-bundle\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:33:30.289893 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.289899    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st4pk\" (UniqueName: \"kubernetes.io/projected/3f754a3b-88f2-4c0a-81b4-873ba820c568-kube-api-access-st4pk\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:33:30.290068 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.289909    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-oauth-serving-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:33:30.290068 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.289919    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-service-ca\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:33:30.290068 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.289927    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:33:30.290068 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.289935    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f754a3b-88f2-4c0a-81b4-873ba820c568-console-serving-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:33:30.838277 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.838241    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f754a3b-88f2-4c0a-81b4-873ba820c568" containerID="fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c" exitCode=0
Jul 21 17:33:30.838729 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.838301    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-557b5d69d4-jzgn8"
Jul 21 17:33:30.838729 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.838322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-557b5d69d4-jzgn8" event={"ID":"3f754a3b-88f2-4c0a-81b4-873ba820c568","Type":"ContainerDied","Data":"fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c"}
Jul 21 17:33:30.838729 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.838395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-557b5d69d4-jzgn8" event={"ID":"3f754a3b-88f2-4c0a-81b4-873ba820c568","Type":"ContainerDied","Data":"c3809a8555b4ec9908bbfbf78a2a66fe08a661079e649a1095a114f61e6d60e6"}
Jul 21 17:33:30.838729 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.838416    2579 scope.go:117] "RemoveContainer" containerID="fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c"
Jul 21 17:33:30.849547 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.849529    2579 scope.go:117] "RemoveContainer" containerID="fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c"
Jul 21 17:33:30.849853 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:33:30.849834    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c\": container with ID starting with fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c not found: ID does not exist" containerID="fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c"
Jul 21 17:33:30.849908 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.849862    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c"} err="failed to get container status \"fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c\": rpc error: code = NotFound desc = could not find container \"fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c\": container with ID starting with fc91f92c78f1b5aee228796c0fd8903ed7a23d5b2aa578e8c3c76dc75e004e3c not found: ID does not exist"
Jul 21 17:33:30.860371 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.860350    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-557b5d69d4-jzgn8"]
Jul 21 17:33:30.866273 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:30.866255    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-557b5d69d4-jzgn8"]
Jul 21 17:33:31.004031 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:31.003994    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f754a3b-88f2-4c0a-81b4-873ba820c568" path="/var/lib/kubelet/pods/3f754a3b-88f2-4c0a-81b4-873ba820c568/volumes"
Jul 21 17:33:32.731785 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:32.731755    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6b69bf598b-f697k"
Jul 21 17:33:33.712997 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.712965    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-5l5zg"]
Jul 21 17:33:33.713290 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.713277    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cec9aa4-08ac-48cb-8893-fe7eedaf16f7" containerName="manager"
Jul 21 17:33:33.713343 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.713292    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cec9aa4-08ac-48cb-8893-fe7eedaf16f7" containerName="manager"
Jul 21 17:33:33.713343 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.713315    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f754a3b-88f2-4c0a-81b4-873ba820c568" containerName="console"
Jul 21 17:33:33.713343 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.713320    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f754a3b-88f2-4c0a-81b4-873ba820c568" containerName="console"
Jul 21 17:33:33.713432 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.713368    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f754a3b-88f2-4c0a-81b4-873ba820c568" containerName="console"
Jul 21 17:33:33.713432 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.713377    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cec9aa4-08ac-48cb-8893-fe7eedaf16f7" containerName="manager"
Jul 21 17:33:33.716547 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.716525    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-5l5zg"
Jul 21 17:33:33.719567 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.719547    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 21 17:33:33.719702 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.719682    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-zhjk4\""
Jul 21 17:33:33.730092 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.730071    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-5l5zg"]
Jul 21 17:33:33.817298 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.817268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q76bv\" (UniqueName: \"kubernetes.io/projected/1389af9a-5961-4272-b30b-83a03a4e70aa-kube-api-access-q76bv\") pod \"model-serving-api-fd65d7d6-5l5zg\" (UID: \"1389af9a-5961-4272-b30b-83a03a4e70aa\") " pod="kserve/model-serving-api-fd65d7d6-5l5zg"
Jul 21 17:33:33.817709 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.817327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1389af9a-5961-4272-b30b-83a03a4e70aa-tls-certs\") pod \"model-serving-api-fd65d7d6-5l5zg\" (UID: \"1389af9a-5961-4272-b30b-83a03a4e70aa\") " pod="kserve/model-serving-api-fd65d7d6-5l5zg"
Jul 21 17:33:33.918743 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.918712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76bv\" (UniqueName: \"kubernetes.io/projected/1389af9a-5961-4272-b30b-83a03a4e70aa-kube-api-access-q76bv\") pod \"model-serving-api-fd65d7d6-5l5zg\" (UID: \"1389af9a-5961-4272-b30b-83a03a4e70aa\") " pod="kserve/model-serving-api-fd65d7d6-5l5zg"
Jul 21 17:33:33.918895 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.918774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1389af9a-5961-4272-b30b-83a03a4e70aa-tls-certs\") pod \"model-serving-api-fd65d7d6-5l5zg\" (UID: \"1389af9a-5961-4272-b30b-83a03a4e70aa\") " pod="kserve/model-serving-api-fd65d7d6-5l5zg"
Jul 21 17:33:33.921080 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.921054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1389af9a-5961-4272-b30b-83a03a4e70aa-tls-certs\") pod \"model-serving-api-fd65d7d6-5l5zg\" (UID: \"1389af9a-5961-4272-b30b-83a03a4e70aa\") " pod="kserve/model-serving-api-fd65d7d6-5l5zg"
Jul 21 17:33:33.928322 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:33.928303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q76bv\" (UniqueName: \"kubernetes.io/projected/1389af9a-5961-4272-b30b-83a03a4e70aa-kube-api-access-q76bv\") pod \"model-serving-api-fd65d7d6-5l5zg\" (UID: \"1389af9a-5961-4272-b30b-83a03a4e70aa\") " pod="kserve/model-serving-api-fd65d7d6-5l5zg"
Jul 21 17:33:34.026788 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:34.026711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-5l5zg"
Jul 21 17:33:34.151283 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:34.151263    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-5l5zg"]
Jul 21 17:33:34.153864 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:33:34.153838    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1389af9a_5961_4272_b30b_83a03a4e70aa.slice/crio-9e59259544007f91418d66562c800fdad8884c734abe606607f4e9ce9f6c782e WatchSource:0}: Error finding container 9e59259544007f91418d66562c800fdad8884c734abe606607f4e9ce9f6c782e: Status 404 returned error can't find the container with id 9e59259544007f91418d66562c800fdad8884c734abe606607f4e9ce9f6c782e
Jul 21 17:33:34.857484 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:34.857449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-5l5zg" event={"ID":"1389af9a-5961-4272-b30b-83a03a4e70aa","Type":"ContainerStarted","Data":"9e59259544007f91418d66562c800fdad8884c734abe606607f4e9ce9f6c782e"}
Jul 21 17:33:36.872092 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:36.872059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-5l5zg" event={"ID":"1389af9a-5961-4272-b30b-83a03a4e70aa","Type":"ContainerStarted","Data":"4dd228f26dd6e2e4ee198bc1807076b5d0845b461cb6ce0ff472ad9d23332cfd"}
Jul 21 17:33:36.872463 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:36.872141    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-5l5zg"
Jul 21 17:33:36.891855 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:36.891810    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-5l5zg" podStartSLOduration=1.619185863 podStartE2EDuration="3.891794896s" totalImagesPullingTime="2.272609033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:33:36.890624781 +0000 UTC m=+442.449423769" watchObservedRunningTime="2026-07-21 17:33:36.891794896 +0000 UTC m=+442.450593879"
Jul 21 17:33:47.880901 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:33:47.880866    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-5l5zg"
Jul 21 17:35:23.049557 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.049525    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"]
Jul 21 17:35:23.053199 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.053169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.056249 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.056226    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-e8d4c-kube-rbac-proxy-sar-config\""
Jul 21 17:35:23.056357 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.056305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:35:23.056357 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.056297    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:35:23.056518 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.056486    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-e8d4c-predictor-serving-cert\""
Jul 21 17:35:23.056563 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.056538    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2v5wd\""
Jul 21 17:35:23.056609 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.056539    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:35:23.056711 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.056684    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:35:23.065235 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.065210    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"]
Jul 21 17:35:23.148846 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.148815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt789\" (UniqueName: \"kubernetes.io/projected/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-api-access-vt789\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.149012 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.148865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.149012 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.148950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.149012 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.148994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.149131 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.149015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.250427 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.250384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.250607 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.250443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.250607 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.250481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.250607 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.250511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.250607 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.250562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt789\" (UniqueName: \"kubernetes.io/projected/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-api-access-vt789\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.250908 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.250886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.251171 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.251146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.251277 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.251200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.252872 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.252854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.257999 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.257978    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt789\" (UniqueName: \"kubernetes.io/projected/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-api-access-vt789\") pod \"isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.363757 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.363733    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:23.492740 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:23.492715    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"]
Jul 21 17:35:23.495195 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:35:23.495170    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729f852e_7f2e_4037_bfc6_bb8c1b8d9001.slice/crio-c807668d65c6107a02ea7d19f1c3d0e120f17703510ba222ff9a6139d8801318 WatchSource:0}: Error finding container c807668d65c6107a02ea7d19f1c3d0e120f17703510ba222ff9a6139d8801318: Status 404 returned error can't find the container with id c807668d65c6107a02ea7d19f1c3d0e120f17703510ba222ff9a6139d8801318
Jul 21 17:35:24.301208 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:24.301165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" event={"ID":"729f852e-7f2e-4037-bfc6-bb8c1b8d9001","Type":"ContainerStarted","Data":"c807668d65c6107a02ea7d19f1c3d0e120f17703510ba222ff9a6139d8801318"}
Jul 21 17:35:27.316130 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:27.316037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" event={"ID":"729f852e-7f2e-4037-bfc6-bb8c1b8d9001","Type":"ContainerStarted","Data":"46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681"}
Jul 21 17:35:31.334487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:31.334456    2579 generic.go:356] "Generic (PLEG): container finished" podID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerID="46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681" exitCode=0
Jul 21 17:35:31.334885 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:31.334527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" event={"ID":"729f852e-7f2e-4037-bfc6-bb8c1b8d9001","Type":"ContainerDied","Data":"46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681"}
Jul 21 17:35:45.403168 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:45.403132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" event={"ID":"729f852e-7f2e-4037-bfc6-bb8c1b8d9001","Type":"ContainerStarted","Data":"effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888"}
Jul 21 17:35:48.421367 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:48.421331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" event={"ID":"729f852e-7f2e-4037-bfc6-bb8c1b8d9001","Type":"ContainerStarted","Data":"a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053"}
Jul 21 17:35:51.436966 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:51.436929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" event={"ID":"729f852e-7f2e-4037-bfc6-bb8c1b8d9001","Type":"ContainerStarted","Data":"10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506"}
Jul 21 17:35:51.437406 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:51.437155    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:51.437406 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:51.437306    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:51.438698 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:51.438638    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:35:51.463953 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:51.463886    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podStartSLOduration=2.286063216 podStartE2EDuration="28.463872398s" totalImagesPullingTime="22.657056224s" totalInitContainerRuntime="3.520752958s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:35:23 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:35:51.463017679 +0000 UTC m=+577.021816666" watchObservedRunningTime="2026-07-21 17:35:51.463872398 +0000 UTC m=+577.022671428"
Jul 21 17:35:52.440889 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:52.440846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:52.441357 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:52.440938    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:35:52.442038 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:52.442007    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:35:52.445975 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:52.445954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:35:53.447633 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:53.447597    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:35:53.448098 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:53.448071    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:35:54.451734 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:54.451694    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:35:54.452111 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:35:54.451987    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:36:04.451793 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:04.451738    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:36:04.452184 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:04.452147    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:36:14.451813 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:14.451776    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:36:14.452306 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:14.452278    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:36:24.452576 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:24.452522    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:36:24.467341 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:24.453077    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:36:34.451797 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:34.451754    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:36:34.452225 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:34.452196    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:36:44.451841 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:44.451739    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:36:44.452304 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:44.452275    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:36:54.452102 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:54.452060    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:36:54.452546 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:36:54.452454    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:04.452703 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:04.452645    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:37:04.453254 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:04.452867    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:37:18.080161 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.080127    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"]
Jul 21 17:37:18.080585 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.080407    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" containerID="cri-o://effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888" gracePeriod=30
Jul 21 17:37:18.080585 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.080434    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" containerID="cri-o://10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506" gracePeriod=30
Jul 21 17:37:18.080731 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.080620    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kube-rbac-proxy" containerID="cri-o://a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053" gracePeriod=30
Jul 21 17:37:18.207514 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.207485    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"]
Jul 21 17:37:18.210998 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.210979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.219130 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.219113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-69373-predictor-serving-cert\""
Jul 21 17:37:18.222898 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.222880    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-69373-kube-rbac-proxy-sar-config\""
Jul 21 17:37:18.229336 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.229314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"]
Jul 21 17:37:18.294930 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.294895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-cabundle-cert\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.295082 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.294956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.295082 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.295005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ef01e8e-1dba-4480-a64a-5edabe558e54-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.295082 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.295036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbrm\" (UniqueName: \"kubernetes.io/projected/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-api-access-tvbrm\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.295082 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.295056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ef01e8e-1dba-4480-a64a-5edabe558e54-proxy-tls\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.327601 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.327570    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"]
Jul 21 17:37:18.331020 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.330978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.334148 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.333977    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-69373-predictor-serving-cert\""
Jul 21 17:37:18.334148 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.333993    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-69373-kube-rbac-proxy-sar-config\""
Jul 21 17:37:18.341339 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.341318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"]
Jul 21 17:37:18.395969 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.395942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.396133 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.395977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ef01e8e-1dba-4480-a64a-5edabe558e54-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.396133 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.396006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbrm\" (UniqueName: \"kubernetes.io/projected/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-api-access-tvbrm\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.396256 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.396128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ef01e8e-1dba-4480-a64a-5edabe558e54-proxy-tls\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.396256 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.396208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-cabundle-cert\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.396438 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.396416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ef01e8e-1dba-4480-a64a-5edabe558e54-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.396693 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.396677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.396827 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.396806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-cabundle-cert\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.398594 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.398574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ef01e8e-1dba-4480-a64a-5edabe558e54-proxy-tls\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.404562 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.404541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbrm\" (UniqueName: \"kubernetes.io/projected/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-api-access-tvbrm\") pod \"isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.497381 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.497343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xphgp\" (UniqueName: \"kubernetes.io/projected/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-api-access-xphgp\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.497557 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.497404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.497557 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.497454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.497557 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.497517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-cabundle-cert\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.497557 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.497544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-proxy-tls\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.520505 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.520477    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:18.598405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.598340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-cabundle-cert\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.598405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.598377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-proxy-tls\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.598562 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.598430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xphgp\" (UniqueName: \"kubernetes.io/projected/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-api-access-xphgp\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.598562 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.598466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.598562 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.598496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.598906 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.598884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.599164 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.599143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-cabundle-cert\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.599299 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.599140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.601207 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.601186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-proxy-tls\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.608373 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.608351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xphgp\" (UniqueName: \"kubernetes.io/projected/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-api-access-xphgp\") pod \"isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.642488 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.642456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:18.653169 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.653140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"]
Jul 21 17:37:18.655058 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:37:18.655032    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef01e8e_1dba_4480_a64a_5edabe558e54.slice/crio-c764c4471a2100072c0f4599c566a51bb52aa71d7f72bd8188b11b564397a4ea WatchSource:0}: Error finding container c764c4471a2100072c0f4599c566a51bb52aa71d7f72bd8188b11b564397a4ea: Status 404 returned error can't find the container with id c764c4471a2100072c0f4599c566a51bb52aa71d7f72bd8188b11b564397a4ea
Jul 21 17:37:18.775705 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.775682    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"]
Jul 21 17:37:18.777814 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:37:18.777790    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fd3a0c_6dbd_4e4c_b616_98b3ac8c2603.slice/crio-14fab372266b2e314a04b9422b1a33120ecac3f3a96f0ca63be1669395f10439 WatchSource:0}: Error finding container 14fab372266b2e314a04b9422b1a33120ecac3f3a96f0ca63be1669395f10439: Status 404 returned error can't find the container with id 14fab372266b2e314a04b9422b1a33120ecac3f3a96f0ca63be1669395f10439
Jul 21 17:37:18.802620 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.802594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" event={"ID":"4ef01e8e-1dba-4480-a64a-5edabe558e54","Type":"ContainerStarted","Data":"e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868"}
Jul 21 17:37:18.802758 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.802687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" event={"ID":"4ef01e8e-1dba-4480-a64a-5edabe558e54","Type":"ContainerStarted","Data":"c764c4471a2100072c0f4599c566a51bb52aa71d7f72bd8188b11b564397a4ea"}
Jul 21 17:37:18.803884 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.803861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" event={"ID":"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603","Type":"ContainerStarted","Data":"14fab372266b2e314a04b9422b1a33120ecac3f3a96f0ca63be1669395f10439"}
Jul 21 17:37:18.806310 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.806288    2579 generic.go:356] "Generic (PLEG): container finished" podID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerID="a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053" exitCode=2
Jul 21 17:37:18.806391 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:18.806337    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" event={"ID":"729f852e-7f2e-4037-bfc6-bb8c1b8d9001","Type":"ContainerDied","Data":"a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053"}
Jul 21 17:37:19.814577 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:19.814535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" event={"ID":"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603","Type":"ContainerStarted","Data":"ef0ee402e252b00f710ddb6216957e7dfd246be4fe356cc30628003b7ddad44a"}
Jul 21 17:37:22.441309 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:22.441268    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 21 17:37:22.837331 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:22.837298    2579 generic.go:356] "Generic (PLEG): container finished" podID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerID="effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888" exitCode=0
Jul 21 17:37:22.837525 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:22.837373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" event={"ID":"729f852e-7f2e-4037-bfc6-bb8c1b8d9001","Type":"ContainerDied","Data":"effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888"}
Jul 21 17:37:22.838896 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:22.838873    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerID="e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868" exitCode=0
Jul 21 17:37:22.839001 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:22.838955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" event={"ID":"4ef01e8e-1dba-4480-a64a-5edabe558e54","Type":"ContainerDied","Data":"e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868"}
Jul 21 17:37:22.840411 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:22.840393    2579 generic.go:356] "Generic (PLEG): container finished" podID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerID="ef0ee402e252b00f710ddb6216957e7dfd246be4fe356cc30628003b7ddad44a" exitCode=0
Jul 21 17:37:22.840484 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:22.840436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" event={"ID":"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603","Type":"ContainerDied","Data":"ef0ee402e252b00f710ddb6216957e7dfd246be4fe356cc30628003b7ddad44a"}
Jul 21 17:37:22.841692 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:22.841677    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:37:23.849687 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:23.849626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" event={"ID":"4ef01e8e-1dba-4480-a64a-5edabe558e54","Type":"ContainerStarted","Data":"58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f"}
Jul 21 17:37:23.850146 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:23.849721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" event={"ID":"4ef01e8e-1dba-4480-a64a-5edabe558e54","Type":"ContainerStarted","Data":"4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5"}
Jul 21 17:37:23.850146 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:23.849946    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:23.872353 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:23.872298    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podStartSLOduration=2.337122561 podStartE2EDuration="5.872278674s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.535156113s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:37:23.870965452 +0000 UTC m=+669.429764437" watchObservedRunningTime="2026-07-21 17:37:23.872278674 +0000 UTC m=+669.431077660"
Jul 21 17:37:24.451927 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:24.451757    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:37:24.452395 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:24.452365    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:24.856278 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:24.856249    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:24.857944 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:24.857912    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:37:25.861721 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:25.861678    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:37:27.441446 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:27.441396    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 21 17:37:30.866558 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:30.866526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:37:30.867243 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:30.867176    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:37:32.441515 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:32.441473    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 21 17:37:32.442021 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:32.441607    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:37:34.452185 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:34.452141    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:37:34.452694 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:34.452637    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:37.442095 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:37.442045    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 21 17:37:40.867101 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:40.867062    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:37:42.441846 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:42.441807    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 21 17:37:42.946694 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:42.946632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" event={"ID":"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603","Type":"ContainerStarted","Data":"81c99ca4210511307d4330324462380d51009779d00805b797dc5959cf633e32"}
Jul 21 17:37:42.946883 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:42.946719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" event={"ID":"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603","Type":"ContainerStarted","Data":"10d34ee3e93f7820fd8892455c52624924321aecba97e79a6b8863a7a3c1af7f"}
Jul 21 17:37:42.947023 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:42.947006    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:42.947181 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:42.947166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:42.948589 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:42.948566    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:37:42.968221 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:42.968160    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podStartSLOduration=2.123663813 podStartE2EDuration="24.968142726s" totalImagesPullingTime="19.203052388s" totalInitContainerRuntime="3.641426525s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:37:42.967053837 +0000 UTC m=+688.525852822" watchObservedRunningTime="2026-07-21 17:37:42.968142726 +0000 UTC m=+688.526941710"
Jul 21 17:37:43.950998 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:43.950957    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:37:44.452354 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:44.452315    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:37:44.452556 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:44.452447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:37:44.452666 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:44.452626    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:44.452769 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:44.452757    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:37:47.441819 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:47.441774    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 21 17:37:48.761806 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.761782    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:37:48.864758 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.864727    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-proxy-tls\") pod \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") "
Jul 21 17:37:48.864879 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.864784    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kserve-provision-location\") pod \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") "
Jul 21 17:37:48.864879 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.864812    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt789\" (UniqueName: \"kubernetes.io/projected/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-api-access-vt789\") pod \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") "
Jul 21 17:37:48.864879 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.864838    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-cabundle-cert\") pod \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") "
Jul 21 17:37:48.864879 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.864874    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-rbac-proxy-sar-config\") pod \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\" (UID: \"729f852e-7f2e-4037-bfc6-bb8c1b8d9001\") "
Jul 21 17:37:48.865190 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.865162    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "729f852e-7f2e-4037-bfc6-bb8c1b8d9001" (UID: "729f852e-7f2e-4037-bfc6-bb8c1b8d9001"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:37:48.865266 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.865226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "729f852e-7f2e-4037-bfc6-bb8c1b8d9001" (UID: "729f852e-7f2e-4037-bfc6-bb8c1b8d9001"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:37:48.865316 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.865269    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "729f852e-7f2e-4037-bfc6-bb8c1b8d9001" (UID: "729f852e-7f2e-4037-bfc6-bb8c1b8d9001"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:37:48.867030 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.867006    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "729f852e-7f2e-4037-bfc6-bb8c1b8d9001" (UID: "729f852e-7f2e-4037-bfc6-bb8c1b8d9001"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:37:48.867030 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.867019    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-api-access-vt789" (OuterVolumeSpecName: "kube-api-access-vt789") pod "729f852e-7f2e-4037-bfc6-bb8c1b8d9001" (UID: "729f852e-7f2e-4037-bfc6-bb8c1b8d9001"). InnerVolumeSpecName "kube-api-access-vt789". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:37:48.955719 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.955629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:37:48.956171 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.956142    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:37:48.966019 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.965999    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:37:48.966102 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.966021    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:37:48.966102 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.966037    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:37:48.966102 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.966049    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt789\" (UniqueName: \"kubernetes.io/projected/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-kube-api-access-vt789\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:37:48.966102 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.966062    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/729f852e-7f2e-4037-bfc6-bb8c1b8d9001-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:37:48.976278 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.976253    2579 generic.go:356] "Generic (PLEG): container finished" podID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerID="10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506" exitCode=0
Jul 21 17:37:48.976361 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.976315    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"
Jul 21 17:37:48.976361 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.976321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" event={"ID":"729f852e-7f2e-4037-bfc6-bb8c1b8d9001","Type":"ContainerDied","Data":"10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506"}
Jul 21 17:37:48.976432 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.976380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz" event={"ID":"729f852e-7f2e-4037-bfc6-bb8c1b8d9001","Type":"ContainerDied","Data":"c807668d65c6107a02ea7d19f1c3d0e120f17703510ba222ff9a6139d8801318"}
Jul 21 17:37:48.976432 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.976412    2579 scope.go:117] "RemoveContainer" containerID="10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506"
Jul 21 17:37:48.984935 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.984799    2579 scope.go:117] "RemoveContainer" containerID="a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053"
Jul 21 17:37:48.992229 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.992212    2579 scope.go:117] "RemoveContainer" containerID="effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888"
Jul 21 17:37:48.999415 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:48.999394    2579 scope.go:117] "RemoveContainer" containerID="46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681"
Jul 21 17:37:49.004934 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:49.004916    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"]
Jul 21 17:37:49.005015 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:49.004946    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8d4c-predictor-6c5c65c584-xc7rz"]
Jul 21 17:37:49.007326 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:49.007312    2579 scope.go:117] "RemoveContainer" containerID="10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506"
Jul 21 17:37:49.007560 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:37:49.007541    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506\": container with ID starting with 10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506 not found: ID does not exist" containerID="10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506"
Jul 21 17:37:49.007631 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:49.007571    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506"} err="failed to get container status \"10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506\": rpc error: code = NotFound desc = could not find container \"10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506\": container with ID starting with 10ff84dc4c9b5c4faa542b38835d52d97e982e5c44f207bbcfed9ad2778a8506 not found: ID does not exist"
Jul 21 17:37:49.007631 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:49.007595    2579 scope.go:117] "RemoveContainer" containerID="a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053"
Jul 21 17:37:49.007902 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:37:49.007885    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053\": container with ID starting with a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053 not found: ID does not exist" containerID="a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053"
Jul 21 17:37:49.007951 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:49.007909    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053"} err="failed to get container status \"a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053\": rpc error: code = NotFound desc = could not find container \"a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053\": container with ID starting with a843c7c9e3355d8a04d8bf19183411aeb43a1d657f94dd402b74d94e130c5053 not found: ID does not exist"
Jul 21 17:37:49.007951 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:49.007931    2579 scope.go:117] "RemoveContainer" containerID="effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888"
Jul 21 17:37:49.008162 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:37:49.008146    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888\": container with ID starting with effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888 not found: ID does not exist" containerID="effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888"
Jul 21 17:37:49.008224 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:49.008169    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888"} err="failed to get container status \"effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888\": rpc error: code = NotFound desc = could not find container \"effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888\": container with ID starting with effcb17e2d08299f4d4543de608db2fd2e1668945cbcdc03cbdb3b3483637888 not found: ID does not exist"
Jul 21 17:37:49.008224 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:49.008189    2579 scope.go:117] "RemoveContainer" containerID="46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681"
Jul 21 17:37:49.008433 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:37:49.008415    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681\": container with ID starting with 46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681 not found: ID does not exist" containerID="46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681"
Jul 21 17:37:49.008474 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:49.008438    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681"} err="failed to get container status \"46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681\": rpc error: code = NotFound desc = could not find container \"46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681\": container with ID starting with 46a7d73313444fc903badf820ded4710e51a52ec911a328c0feb7baf6b65d681 not found: ID does not exist"
Jul 21 17:37:50.867503 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:50.867469    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:37:51.002858 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:51.002826    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" path="/var/lib/kubelet/pods/729f852e-7f2e-4037-bfc6-bb8c1b8d9001/volumes"
Jul 21 17:37:58.956902 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:37:58.956863    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:38:00.867884 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:38:00.867841    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:38:08.956543 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:38:08.956456    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:38:10.867852 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:38:10.867811    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:38:18.956845 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:38:18.956804    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:38:20.867090 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:38:20.867045    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:38:28.956364 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:38:28.956327    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:38:30.867864 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:38:30.867826    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:38:38.956818 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:38:38.956787    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:38:40.867793 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:38:40.867769    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:39:08.429804 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.429765    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"]
Jul 21 17:39:08.430386 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.430058    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" containerID="cri-o://4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5" gracePeriod=30
Jul 21 17:39:08.430386 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.430098    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kube-rbac-proxy" containerID="cri-o://58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f" gracePeriod=30
Jul 21 17:39:08.490117 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490083    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"]
Jul 21 17:39:08.490439 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490427    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kube-rbac-proxy"
Jul 21 17:39:08.490494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490440    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kube-rbac-proxy"
Jul 21 17:39:08.490494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490450    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="storage-initializer"
Jul 21 17:39:08.490494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490456    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="storage-initializer"
Jul 21 17:39:08.490494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490464    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container"
Jul 21 17:39:08.490494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490469    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container"
Jul 21 17:39:08.490494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490474    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent"
Jul 21 17:39:08.490494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490479    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent"
Jul 21 17:39:08.490763 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490554    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="agent"
Jul 21 17:39:08.490763 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490561    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kserve-container"
Jul 21 17:39:08.490763 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.490568    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="729f852e-7f2e-4037-bfc6-bb8c1b8d9001" containerName="kube-rbac-proxy"
Jul 21 17:39:08.493574 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.493557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.496400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.496376    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-3ad06-kube-rbac-proxy-sar-config\""
Jul 21 17:39:08.496505 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.496451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-3ad06-predictor-serving-cert\""
Jul 21 17:39:08.505372 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.505350    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"]
Jul 21 17:39:08.634758 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.634718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.634943 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.634773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ndl\" (UniqueName: \"kubernetes.io/projected/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-api-access-l7ndl\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.634943 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.634858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.634943 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.634900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54dbcf02-71e1-4da0-9b91-184e0f11697b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.634943 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.634936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54dbcf02-71e1-4da0-9b91-184e0f11697b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.672802 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.672773    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"]
Jul 21 17:39:08.673025 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.672990    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container" containerID="cri-o://10d34ee3e93f7820fd8892455c52624924321aecba97e79a6b8863a7a3c1af7f" gracePeriod=30
Jul 21 17:39:08.673068 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.673027    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kube-rbac-proxy" containerID="cri-o://81c99ca4210511307d4330324462380d51009779d00805b797dc5959cf633e32" gracePeriod=30
Jul 21 17:39:08.735479 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.735453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.735610 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.735491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54dbcf02-71e1-4da0-9b91-184e0f11697b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.735610 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.735519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54dbcf02-71e1-4da0-9b91-184e0f11697b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.735610 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.735559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.735610 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.735581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ndl\" (UniqueName: \"kubernetes.io/projected/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-api-access-l7ndl\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.735827 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:39:08.735717    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-3ad06-predictor-serving-cert" not found
Jul 21 17:39:08.735827 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:39:08.735779    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54dbcf02-71e1-4da0-9b91-184e0f11697b-proxy-tls podName:54dbcf02-71e1-4da0-9b91-184e0f11697b nodeName:}" failed. No retries permitted until 2026-07-21 17:39:09.235758298 +0000 UTC m=+774.794557263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/54dbcf02-71e1-4da0-9b91-184e0f11697b-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" (UID: "54dbcf02-71e1-4da0-9b91-184e0f11697b") : secret "isvc-sklearn-graph-raw-hpa-3ad06-predictor-serving-cert" not found
Jul 21 17:39:08.735917 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.735881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54dbcf02-71e1-4da0-9b91-184e0f11697b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.736160 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.736144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.736300 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.736279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.744473 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.744455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ndl\" (UniqueName: \"kubernetes.io/projected/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-api-access-l7ndl\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:08.951244 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.951158    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 21 17:39:08.956144 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:08.956114    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:39:09.240410 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:09.240327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54dbcf02-71e1-4da0-9b91-184e0f11697b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:09.242726 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:09.242707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54dbcf02-71e1-4da0-9b91-184e0f11697b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:09.338688 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:09.338640    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerID="58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f" exitCode=2
Jul 21 17:39:09.338860 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:09.338687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" event={"ID":"4ef01e8e-1dba-4480-a64a-5edabe558e54","Type":"ContainerDied","Data":"58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f"}
Jul 21 17:39:09.340826 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:09.340806    2579 generic.go:356] "Generic (PLEG): container finished" podID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerID="81c99ca4210511307d4330324462380d51009779d00805b797dc5959cf633e32" exitCode=2
Jul 21 17:39:09.340912 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:09.340877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" event={"ID":"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603","Type":"ContainerDied","Data":"81c99ca4210511307d4330324462380d51009779d00805b797dc5959cf633e32"}
Jul 21 17:39:09.405837 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:09.405802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:09.533497 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:09.533474    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"]
Jul 21 17:39:09.535588 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:39:09.535559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dbcf02_71e1_4da0_9b91_184e0f11697b.slice/crio-d181c3c925d049ef74fbb8ffa69024e393ba20de62bd30845e8d33b2b856e35e WatchSource:0}: Error finding container d181c3c925d049ef74fbb8ffa69024e393ba20de62bd30845e8d33b2b856e35e: Status 404 returned error can't find the container with id d181c3c925d049ef74fbb8ffa69024e393ba20de62bd30845e8d33b2b856e35e
Jul 21 17:39:10.347025 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:10.346994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" event={"ID":"54dbcf02-71e1-4da0-9b91-184e0f11697b","Type":"ContainerStarted","Data":"bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068"}
Jul 21 17:39:10.347192 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:10.347052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" event={"ID":"54dbcf02-71e1-4da0-9b91-184e0f11697b","Type":"ContainerStarted","Data":"d181c3c925d049ef74fbb8ffa69024e393ba20de62bd30845e8d33b2b856e35e"}
Jul 21 17:39:10.862449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:10.862403    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 21 17:39:10.868067 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:10.868037    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:39:12.359190 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.359060    2579 generic.go:356] "Generic (PLEG): container finished" podID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerID="10d34ee3e93f7820fd8892455c52624924321aecba97e79a6b8863a7a3c1af7f" exitCode=0
Jul 21 17:39:12.359190 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.359088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" event={"ID":"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603","Type":"ContainerDied","Data":"10d34ee3e93f7820fd8892455c52624924321aecba97e79a6b8863a7a3c1af7f"}
Jul 21 17:39:12.425196 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.425153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:39:12.464931 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.464904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-proxy-tls\") pod \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") "
Jul 21 17:39:12.465093 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.464949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-rbac-proxy-sar-config\") pod \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") "
Jul 21 17:39:12.465093 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.464981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-cabundle-cert\") pod \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") "
Jul 21 17:39:12.465093 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.465003    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kserve-provision-location\") pod \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") "
Jul 21 17:39:12.465093 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.465027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xphgp\" (UniqueName: \"kubernetes.io/projected/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-api-access-xphgp\") pod \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\" (UID: \"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603\") "
Jul 21 17:39:12.465385 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.465361    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" (UID: "06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:39:12.465457 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.465396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" (UID: "06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:39:12.465457 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.465404    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" (UID: "06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:39:12.467061 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.467034    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" (UID: "06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:39:12.467153 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.467134    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-api-access-xphgp" (OuterVolumeSpecName: "kube-api-access-xphgp") pod "06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" (UID: "06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603"). InnerVolumeSpecName "kube-api-access-xphgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:39:12.565801 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.565770    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:39:12.565801 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.565795    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:39:12.565801 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.565803    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:39:12.566057 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.565812    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xphgp\" (UniqueName: \"kubernetes.io/projected/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-kube-api-access-xphgp\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:39:12.566057 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:12.565822    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:39:13.177785 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.177761    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:39:13.272467 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.272409    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ef01e8e-1dba-4480-a64a-5edabe558e54-proxy-tls\") pod \"4ef01e8e-1dba-4480-a64a-5edabe558e54\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") "
Jul 21 17:39:13.272467 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.272449    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-rbac-proxy-sar-config\") pod \"4ef01e8e-1dba-4480-a64a-5edabe558e54\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") "
Jul 21 17:39:13.272608 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.272476    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvbrm\" (UniqueName: \"kubernetes.io/projected/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-api-access-tvbrm\") pod \"4ef01e8e-1dba-4480-a64a-5edabe558e54\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") "
Jul 21 17:39:13.272608 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.272503    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ef01e8e-1dba-4480-a64a-5edabe558e54-kserve-provision-location\") pod \"4ef01e8e-1dba-4480-a64a-5edabe558e54\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") "
Jul 21 17:39:13.272608 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.272553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-cabundle-cert\") pod \"4ef01e8e-1dba-4480-a64a-5edabe558e54\" (UID: \"4ef01e8e-1dba-4480-a64a-5edabe558e54\") "
Jul 21 17:39:13.272803 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.272782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ef01e8e-1dba-4480-a64a-5edabe558e54" (UID: "4ef01e8e-1dba-4480-a64a-5edabe558e54"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:39:13.272871 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.272842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef01e8e-1dba-4480-a64a-5edabe558e54-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ef01e8e-1dba-4480-a64a-5edabe558e54" (UID: "4ef01e8e-1dba-4480-a64a-5edabe558e54"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:39:13.272945 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.272926    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ef01e8e-1dba-4480-a64a-5edabe558e54" (UID: "4ef01e8e-1dba-4480-a64a-5edabe558e54"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:39:13.274568 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.274547    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ef01e8e-1dba-4480-a64a-5edabe558e54-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ef01e8e-1dba-4480-a64a-5edabe558e54" (UID: "4ef01e8e-1dba-4480-a64a-5edabe558e54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:39:13.274620 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.274592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-api-access-tvbrm" (OuterVolumeSpecName: "kube-api-access-tvbrm") pod "4ef01e8e-1dba-4480-a64a-5edabe558e54" (UID: "4ef01e8e-1dba-4480-a64a-5edabe558e54"). InnerVolumeSpecName "kube-api-access-tvbrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:39:13.365664 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.365621    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerID="4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5" exitCode=0
Jul 21 17:39:13.366085 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.365690    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"
Jul 21 17:39:13.366085 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.365758    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" event={"ID":"4ef01e8e-1dba-4480-a64a-5edabe558e54","Type":"ContainerDied","Data":"4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5"}
Jul 21 17:39:13.366085 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.365837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs" event={"ID":"4ef01e8e-1dba-4480-a64a-5edabe558e54","Type":"ContainerDied","Data":"c764c4471a2100072c0f4599c566a51bb52aa71d7f72bd8188b11b564397a4ea"}
Jul 21 17:39:13.366085 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.365875    2579 scope.go:117] "RemoveContainer" containerID="58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f"
Jul 21 17:39:13.367935 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.367918    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"
Jul 21 17:39:13.368050 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.367934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w" event={"ID":"06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603","Type":"ContainerDied","Data":"14fab372266b2e314a04b9422b1a33120ecac3f3a96f0ca63be1669395f10439"}
Jul 21 17:39:13.373835 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.373812    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:39:13.373951 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.373838    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvbrm\" (UniqueName: \"kubernetes.io/projected/4ef01e8e-1dba-4480-a64a-5edabe558e54-kube-api-access-tvbrm\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:39:13.373951 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.373870    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ef01e8e-1dba-4480-a64a-5edabe558e54-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:39:13.373951 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.373885    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ef01e8e-1dba-4480-a64a-5edabe558e54-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:39:13.373951 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.373898    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ef01e8e-1dba-4480-a64a-5edabe558e54-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:39:13.375514 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.375500    2579 scope.go:117] "RemoveContainer" containerID="4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5"
Jul 21 17:39:13.383748 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.383733    2579 scope.go:117] "RemoveContainer" containerID="e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868"
Jul 21 17:39:13.389926 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.389904    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"]
Jul 21 17:39:13.391061 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.391048    2579 scope.go:117] "RemoveContainer" containerID="58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f"
Jul 21 17:39:13.391310 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:39:13.391286    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f\": container with ID starting with 58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f not found: ID does not exist" containerID="58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f"
Jul 21 17:39:13.391368 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.391317    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f"} err="failed to get container status \"58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f\": rpc error: code = NotFound desc = could not find container \"58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f\": container with ID starting with 58acf4d812cf37bfe256761553594f767635e0456af45fa96d93bcba8d16823f not found: ID does not exist"
Jul 21 17:39:13.391368 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.391335    2579 scope.go:117] "RemoveContainer" containerID="4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5"
Jul 21 17:39:13.391561 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:39:13.391544    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5\": container with ID starting with 4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5 not found: ID does not exist" containerID="4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5"
Jul 21 17:39:13.391623 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.391570    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5"} err="failed to get container status \"4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5\": rpc error: code = NotFound desc = could not find container \"4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5\": container with ID starting with 4c132bb5abc0b74fd455a04759697c718582925e485e525dc22c02dcb5d122d5 not found: ID does not exist"
Jul 21 17:39:13.391623 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.391594    2579 scope.go:117] "RemoveContainer" containerID="e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868"
Jul 21 17:39:13.391823 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:39:13.391807    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868\": container with ID starting with e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868 not found: ID does not exist" containerID="e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868"
Jul 21 17:39:13.391868 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.391828    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868"} err="failed to get container status \"e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868\": rpc error: code = NotFound desc = could not find container \"e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868\": container with ID starting with e1b3cb60efe4ca6f1d230d39dd1559e5f0a522d69858434de18a876a0cd09868 not found: ID does not exist"
Jul 21 17:39:13.391868 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.391842    2579 scope.go:117] "RemoveContainer" containerID="81c99ca4210511307d4330324462380d51009779d00805b797dc5959cf633e32"
Jul 21 17:39:13.398154 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.398126    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-69373-predictor-567f6dfdd8-c46cs"]
Jul 21 17:39:13.402066 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.402046    2579 scope.go:117] "RemoveContainer" containerID="10d34ee3e93f7820fd8892455c52624924321aecba97e79a6b8863a7a3c1af7f"
Jul 21 17:39:13.409492 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.409473    2579 scope.go:117] "RemoveContainer" containerID="ef0ee402e252b00f710ddb6216957e7dfd246be4fe356cc30628003b7ddad44a"
Jul 21 17:39:13.411111 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.411089    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"]
Jul 21 17:39:13.415141 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:13.415122    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-69373-predictor-5dd484c6b9-8f67w"]
Jul 21 17:39:14.377533 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:14.377500    2579 generic.go:356] "Generic (PLEG): container finished" podID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerID="bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068" exitCode=0
Jul 21 17:39:14.377909 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:14.377578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" event={"ID":"54dbcf02-71e1-4da0-9b91-184e0f11697b","Type":"ContainerDied","Data":"bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068"}
Jul 21 17:39:15.009460 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:15.009429    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" path="/var/lib/kubelet/pods/06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603/volumes"
Jul 21 17:39:15.009949 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:15.009936    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" path="/var/lib/kubelet/pods/4ef01e8e-1dba-4480-a64a-5edabe558e54/volumes"
Jul 21 17:39:15.383981 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:15.383947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" event={"ID":"54dbcf02-71e1-4da0-9b91-184e0f11697b","Type":"ContainerStarted","Data":"330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773"}
Jul 21 17:39:15.384427 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:15.384019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" event={"ID":"54dbcf02-71e1-4da0-9b91-184e0f11697b","Type":"ContainerStarted","Data":"6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce"}
Jul 21 17:39:15.384427 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:15.384296    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:15.384427 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:15.384347    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:15.385780 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:15.385755    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:39:15.407682 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:15.407624    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podStartSLOduration=3.611172158 podStartE2EDuration="7.407611997s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.796439839s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:39:15.405684767 +0000 UTC m=+780.964483751" watchObservedRunningTime="2026-07-21 17:39:15.407611997 +0000 UTC m=+780.966410980"
Jul 21 17:39:16.388320 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:16.388283    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:39:21.392863 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:21.392838    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:39:21.393322 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:21.393296    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:39:31.393500 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:31.393454    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:39:41.393382 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:41.393286    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:39:51.393292 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:39:51.393251    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:40:01.393441 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:01.393394    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:40:11.394270 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:11.394231    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:40:21.393341 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:21.393297    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:40:31.393805 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:31.393773    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:40:58.798023 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.797991    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"]
Jul 21 17:40:58.798511 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.798261    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" containerID="cri-o://6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce" gracePeriod=30
Jul 21 17:40:58.798511 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.798294    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kube-rbac-proxy" containerID="cri-o://330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773" gracePeriod=30
Jul 21 17:40:58.825796 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.825765    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"]
Jul 21 17:40:58.826262 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826247    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="storage-initializer"
Jul 21 17:40:58.826304 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826266    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="storage-initializer"
Jul 21 17:40:58.826304 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826278    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kube-rbac-proxy"
Jul 21 17:40:58.826304 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826287    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kube-rbac-proxy"
Jul 21 17:40:58.826405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826303    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container"
Jul 21 17:40:58.826405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826312    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container"
Jul 21 17:40:58.826405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826326    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container"
Jul 21 17:40:58.826405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826334    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container"
Jul 21 17:40:58.826405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826347    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="storage-initializer"
Jul 21 17:40:58.826405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826355    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="storage-initializer"
Jul 21 17:40:58.826405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826378    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kube-rbac-proxy"
Jul 21 17:40:58.826405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826385    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kube-rbac-proxy"
Jul 21 17:40:58.826627 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826461    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kserve-container"
Jul 21 17:40:58.826627 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826474    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef01e8e-1dba-4480-a64a-5edabe558e54" containerName="kube-rbac-proxy"
Jul 21 17:40:58.826627 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826486    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kserve-container"
Jul 21 17:40:58.826627 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.826496    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="06fd3a0c-6dbd-4e4c-b616-98b3ac8c2603" containerName="kube-rbac-proxy"
Jul 21 17:40:58.830046 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.830029    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:58.834772 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.834753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4098a-kube-rbac-proxy-sar-config\""
Jul 21 17:40:58.834876 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.834758    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4098a-predictor-serving-cert\""
Jul 21 17:40:58.843463 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.843440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"]
Jul 21 17:40:58.911493 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.911464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-proxy-tls\") pod \"message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:58.911642 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.911499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jlhn\" (UniqueName: \"kubernetes.io/projected/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-api-access-5jlhn\") pod \"message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:58.911642 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:58.911528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:59.012192 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.012157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-proxy-tls\") pod \"message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:59.012377 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.012200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jlhn\" (UniqueName: \"kubernetes.io/projected/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-api-access-5jlhn\") pod \"message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:59.012377 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.012240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:59.013011 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.012982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:59.014742 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.014721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-proxy-tls\") pod \"message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:59.022822 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.022786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jlhn\" (UniqueName: \"kubernetes.io/projected/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-api-access-5jlhn\") pod \"message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:59.140467 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.140429    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:40:59.276647 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.276622    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"]
Jul 21 17:40:59.278802 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:40:59.278775    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f080ef3_9e5b_4f7f_b17d_e8ff69f06b4c.slice/crio-797c42f261d02e496713e461ed251c7432503c2697cd1b558347059c58c6cd0f WatchSource:0}: Error finding container 797c42f261d02e496713e461ed251c7432503c2697cd1b558347059c58c6cd0f: Status 404 returned error can't find the container with id 797c42f261d02e496713e461ed251c7432503c2697cd1b558347059c58c6cd0f
Jul 21 17:40:59.802550 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.802516    2579 generic.go:356] "Generic (PLEG): container finished" podID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerID="330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773" exitCode=2
Jul 21 17:40:59.802953 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.802591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" event={"ID":"54dbcf02-71e1-4da0-9b91-184e0f11697b","Type":"ContainerDied","Data":"330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773"}
Jul 21 17:40:59.803731 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:40:59.803707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg" event={"ID":"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c","Type":"ContainerStarted","Data":"797c42f261d02e496713e461ed251c7432503c2697cd1b558347059c58c6cd0f"}
Jul 21 17:41:00.810245 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:00.810160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg" event={"ID":"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c","Type":"ContainerStarted","Data":"5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2"}
Jul 21 17:41:00.810245 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:00.810222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg" event={"ID":"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c","Type":"ContainerStarted","Data":"0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36"}
Jul 21 17:41:00.810669 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:00.810252    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:41:00.830811 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:00.830762    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg" podStartSLOduration=1.6300924860000001 podStartE2EDuration="2.830747105s" totalImagesPullingTime="1.200654619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:41:00.828542852 +0000 UTC m=+886.387341862" watchObservedRunningTime="2026-07-21 17:41:00.830747105 +0000 UTC m=+886.389546090"
Jul 21 17:41:01.389584 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:01.389545    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 21 17:41:01.393915 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:01.393888    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:41:01.815464 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:01.815382    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:41:01.817293 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:01.817269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:41:03.649741 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.649715    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:41:03.755146 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.755054    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-cabundle-cert\") pod \"54dbcf02-71e1-4da0-9b91-184e0f11697b\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") "
Jul 21 17:41:03.755321 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.755170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54dbcf02-71e1-4da0-9b91-184e0f11697b-kserve-provision-location\") pod \"54dbcf02-71e1-4da0-9b91-184e0f11697b\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") "
Jul 21 17:41:03.755321 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.755199    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54dbcf02-71e1-4da0-9b91-184e0f11697b-proxy-tls\") pod \"54dbcf02-71e1-4da0-9b91-184e0f11697b\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") "
Jul 21 17:41:03.755321 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.755239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-rbac-proxy-sar-config\") pod \"54dbcf02-71e1-4da0-9b91-184e0f11697b\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") "
Jul 21 17:41:03.755321 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.755274    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7ndl\" (UniqueName: \"kubernetes.io/projected/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-api-access-l7ndl\") pod \"54dbcf02-71e1-4da0-9b91-184e0f11697b\" (UID: \"54dbcf02-71e1-4da0-9b91-184e0f11697b\") "
Jul 21 17:41:03.755518 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.755479    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dbcf02-71e1-4da0-9b91-184e0f11697b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54dbcf02-71e1-4da0-9b91-184e0f11697b" (UID: "54dbcf02-71e1-4da0-9b91-184e0f11697b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:41:03.755561 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.755510    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "54dbcf02-71e1-4da0-9b91-184e0f11697b" (UID: "54dbcf02-71e1-4da0-9b91-184e0f11697b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:41:03.755637 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.755612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "54dbcf02-71e1-4da0-9b91-184e0f11697b" (UID: "54dbcf02-71e1-4da0-9b91-184e0f11697b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:41:03.757259 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.757231    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54dbcf02-71e1-4da0-9b91-184e0f11697b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54dbcf02-71e1-4da0-9b91-184e0f11697b" (UID: "54dbcf02-71e1-4da0-9b91-184e0f11697b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:41:03.757362 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.757282    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-api-access-l7ndl" (OuterVolumeSpecName: "kube-api-access-l7ndl") pod "54dbcf02-71e1-4da0-9b91-184e0f11697b" (UID: "54dbcf02-71e1-4da0-9b91-184e0f11697b"). InnerVolumeSpecName "kube-api-access-l7ndl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:41:03.827114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.827077    2579 generic.go:356] "Generic (PLEG): container finished" podID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerID="6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce" exitCode=0
Jul 21 17:41:03.827277 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.827147    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"
Jul 21 17:41:03.827277 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.827163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" event={"ID":"54dbcf02-71e1-4da0-9b91-184e0f11697b","Type":"ContainerDied","Data":"6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce"}
Jul 21 17:41:03.827277 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.827228    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c" event={"ID":"54dbcf02-71e1-4da0-9b91-184e0f11697b","Type":"ContainerDied","Data":"d181c3c925d049ef74fbb8ffa69024e393ba20de62bd30845e8d33b2b856e35e"}
Jul 21 17:41:03.827277 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.827255    2579 scope.go:117] "RemoveContainer" containerID="330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773"
Jul 21 17:41:03.835975 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.835957    2579 scope.go:117] "RemoveContainer" containerID="6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce"
Jul 21 17:41:03.843528 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.843509    2579 scope.go:117] "RemoveContainer" containerID="bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068"
Jul 21 17:41:03.850884 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.850858    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"]
Jul 21 17:41:03.851209 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.851195    2579 scope.go:117] "RemoveContainer" containerID="330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773"
Jul 21 17:41:03.851493 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:41:03.851468    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773\": container with ID starting with 330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773 not found: ID does not exist" containerID="330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773"
Jul 21 17:41:03.851539 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.851500    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773"} err="failed to get container status \"330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773\": rpc error: code = NotFound desc = could not find container \"330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773\": container with ID starting with 330d39974d732f0de071ba60c4c347779df1411d9d7933c63d569d08ed940773 not found: ID does not exist"
Jul 21 17:41:03.851539 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.851516    2579 scope.go:117] "RemoveContainer" containerID="6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce"
Jul 21 17:41:03.851788 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:41:03.851764    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce\": container with ID starting with 6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce not found: ID does not exist" containerID="6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce"
Jul 21 17:41:03.851890 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.851794    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce"} err="failed to get container status \"6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce\": rpc error: code = NotFound desc = could not find container \"6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce\": container with ID starting with 6da833f2b7fb1f9efe6177fd9a98f0e3edc9b0e7ed5eb52ed3921e3a4ac1d8ce not found: ID does not exist"
Jul 21 17:41:03.851890 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.851810    2579 scope.go:117] "RemoveContainer" containerID="bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068"
Jul 21 17:41:03.852083 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:41:03.852066    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068\": container with ID starting with bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068 not found: ID does not exist" containerID="bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068"
Jul 21 17:41:03.852124 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.852089    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068"} err="failed to get container status \"bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068\": rpc error: code = NotFound desc = could not find container \"bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068\": container with ID starting with bb572614ce22e21dc9e133ee5b6479680165cf3503d021c798a86c965a257068 not found: ID does not exist"
Jul 21 17:41:03.856494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.856469    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54dbcf02-71e1-4da0-9b91-184e0f11697b-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:41:03.856494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.856495    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54dbcf02-71e1-4da0-9b91-184e0f11697b-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:41:03.856625 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.856506    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:41:03.856625 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.856514    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7ndl\" (UniqueName: \"kubernetes.io/projected/54dbcf02-71e1-4da0-9b91-184e0f11697b-kube-api-access-l7ndl\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:41:03.856625 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.856523    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54dbcf02-71e1-4da0-9b91-184e0f11697b-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:41:03.859513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:03.859491    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3ad06-predictor-87ff8fc76-nrc9c"]
Jul 21 17:41:05.003746 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:05.003707    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" path="/var/lib/kubelet/pods/54dbcf02-71e1-4da0-9b91-184e0f11697b/volumes"
Jul 21 17:41:08.832081 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:08.832001    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:41:18.870002 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.869962    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"]
Jul 21 17:41:18.871246 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.871219    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container"
Jul 21 17:41:18.871397 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.871385    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container"
Jul 21 17:41:18.871548 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.871536    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kube-rbac-proxy"
Jul 21 17:41:18.871622 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.871613    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kube-rbac-proxy"
Jul 21 17:41:18.871727 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.871715    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="storage-initializer"
Jul 21 17:41:18.871798 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.871790    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="storage-initializer"
Jul 21 17:41:18.872038 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.872025    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kube-rbac-proxy"
Jul 21 17:41:18.872131 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.872122    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54dbcf02-71e1-4da0-9b91-184e0f11697b" containerName="kserve-container"
Jul 21 17:41:18.878111 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.878089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:18.880876 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.880849    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:41:18.881030 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.880851    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:41:18.881030 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.880895    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4098a-kube-rbac-proxy-sar-config\""
Jul 21 17:41:18.881030 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.880960    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4098a-predictor-serving-cert\""
Jul 21 17:41:18.884514 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.884493    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"]
Jul 21 17:41:18.979276 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.979247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-cabundle-cert\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:18.979421 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.979284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pcfl\" (UniqueName: \"kubernetes.io/projected/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-api-access-6pcfl\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:18.979421 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.979341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kserve-provision-location\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:18.979421 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.979374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:18.979527 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:18.979438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b45d18-c05d-420d-9ef8-4da31c1fc48f-proxy-tls\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.080353 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.080321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kserve-provision-location\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.080513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.080366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.080513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.080397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b45d18-c05d-420d-9ef8-4da31c1fc48f-proxy-tls\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.080513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.080468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-cabundle-cert\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.080513 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.080498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcfl\" (UniqueName: \"kubernetes.io/projected/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-api-access-6pcfl\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.080823 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.080802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kserve-provision-location\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.081143 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.081115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-cabundle-cert\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.081143 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.081124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.082871 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.082853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b45d18-c05d-420d-9ef8-4da31c1fc48f-proxy-tls\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.088846 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.088816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pcfl\" (UniqueName: \"kubernetes.io/projected/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-api-access-6pcfl\") pod \"isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.189900 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.189823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:19.323047 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.323023    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"]
Jul 21 17:41:19.325501 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:41:19.325472    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b45d18_c05d_420d_9ef8_4da31c1fc48f.slice/crio-7f918d9d74796163d4e3352ed18f0989377ee6a9f0cc3a20959fb602ef72091f WatchSource:0}: Error finding container 7f918d9d74796163d4e3352ed18f0989377ee6a9f0cc3a20959fb602ef72091f: Status 404 returned error can't find the container with id 7f918d9d74796163d4e3352ed18f0989377ee6a9f0cc3a20959fb602ef72091f
Jul 21 17:41:19.897209 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.897169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" event={"ID":"45b45d18-c05d-420d-9ef8-4da31c1fc48f","Type":"ContainerStarted","Data":"8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6"}
Jul 21 17:41:19.897614 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:19.897234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" event={"ID":"45b45d18-c05d-420d-9ef8-4da31c1fc48f","Type":"ContainerStarted","Data":"7f918d9d74796163d4e3352ed18f0989377ee6a9f0cc3a20959fb602ef72091f"}
Jul 21 17:41:23.915279 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:23.915239    2579 generic.go:356] "Generic (PLEG): container finished" podID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerID="8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6" exitCode=0
Jul 21 17:41:23.915828 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:23.915298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" event={"ID":"45b45d18-c05d-420d-9ef8-4da31c1fc48f","Type":"ContainerDied","Data":"8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6"}
Jul 21 17:41:24.923244 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:24.923204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" event={"ID":"45b45d18-c05d-420d-9ef8-4da31c1fc48f","Type":"ContainerStarted","Data":"6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1"}
Jul 21 17:41:24.925302 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:24.923324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" event={"ID":"45b45d18-c05d-420d-9ef8-4da31c1fc48f","Type":"ContainerStarted","Data":"5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb"}
Jul 21 17:41:24.925302 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:24.923368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" event={"ID":"45b45d18-c05d-420d-9ef8-4da31c1fc48f","Type":"ContainerStarted","Data":"6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c"}
Jul 21 17:41:24.925302 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:24.923622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:24.947034 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:24.946981    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podStartSLOduration=3.40060187 podStartE2EDuration="6.94696551s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.54636364s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:41:24.945554794 +0000 UTC m=+910.504353804" watchObservedRunningTime="2026-07-21 17:41:24.94696551 +0000 UTC m=+910.505764495"
Jul 21 17:41:25.927706 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:25.927677    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:25.928106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:25.927752    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:25.929318 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:25.929287    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:41:25.930013 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:25.929991    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:41:26.932023 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:26.931974    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:41:26.932477 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:26.932455    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:41:31.936782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:31.936754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:41:31.937472 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:31.937444    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:41:31.937776 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:31.937748    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:41:41.938049 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:41.938005    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:41:41.938562 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:41.938536    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:41:51.937587 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:51.937545    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:41:51.938065 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:41:51.938042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:42:01.938165 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:01.938123    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:42:01.938672 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:01.938616    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:42:11.937693 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:11.937638    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:42:11.938245 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:11.938124    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:42:21.937457 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:21.937418    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:42:21.937992 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:21.937889    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:42:31.937561 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:31.937521    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:42:31.938042 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:31.937956    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:42:41.938704 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:41.938612    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:42:41.939072 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:41.938859    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:42:53.882203 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:53.882168    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg_1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c/kserve-container/0.log"
Jul 21 17:42:54.108931 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.108901    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"]
Jul 21 17:42:54.109221 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.109190    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" containerID="cri-o://6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c" gracePeriod=30
Jul 21 17:42:54.109521 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.109190    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" containerID="cri-o://6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1" gracePeriod=30
Jul 21 17:42:54.109629 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.109241    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kube-rbac-proxy" containerID="cri-o://5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb" gracePeriod=30
Jul 21 17:42:54.179243 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.179174    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"]
Jul 21 17:42:54.182751 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.182733    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.185415 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.185400    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-90cae-predictor-serving-cert\""
Jul 21 17:42:54.185512 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.185493    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-90cae-kube-rbac-proxy-sar-config\""
Jul 21 17:42:54.193775 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.193756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"]
Jul 21 17:42:54.215930 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.215900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz24n\" (UniqueName: \"kubernetes.io/projected/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-api-access-qz24n\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.216051 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.215965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-cabundle-cert\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.216051 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.215998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.216051 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.216041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b15d8fb-fd37-4092-a82d-855f29707a03-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.216229 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.216114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b15d8fb-fd37-4092-a82d-855f29707a03-proxy-tls\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.221059 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.221027    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"]
Jul 21 17:42:54.221318 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.221290    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg" podUID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" containerName="kserve-container" containerID="cri-o://0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36" gracePeriod=30
Jul 21 17:42:54.221418 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.221319    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg" podUID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" containerName="kube-rbac-proxy" containerID="cri-o://5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2" gracePeriod=30
Jul 21 17:42:54.317190 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.317162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz24n\" (UniqueName: \"kubernetes.io/projected/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-api-access-qz24n\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.317351 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.317212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-cabundle-cert\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.317351 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.317238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.317351 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.317274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b15d8fb-fd37-4092-a82d-855f29707a03-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.317351 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.317337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b15d8fb-fd37-4092-a82d-855f29707a03-proxy-tls\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.317597 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:42:54.317432    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-serving-cert: secret "isvc-sklearn-scale-raw-90cae-predictor-serving-cert" not found
Jul 21 17:42:54.317597 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:42:54.317484    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b15d8fb-fd37-4092-a82d-855f29707a03-proxy-tls podName:7b15d8fb-fd37-4092-a82d-855f29707a03 nodeName:}" failed. No retries permitted until 2026-07-21 17:42:54.817466015 +0000 UTC m=+1000.376264977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7b15d8fb-fd37-4092-a82d-855f29707a03-proxy-tls") pod "isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" (UID: "7b15d8fb-fd37-4092-a82d-855f29707a03") : secret "isvc-sklearn-scale-raw-90cae-predictor-serving-cert" not found
Jul 21 17:42:54.317775 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.317747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b15d8fb-fd37-4092-a82d-855f29707a03-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.318028 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.318009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-cabundle-cert\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.318127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.318047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.320346 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.320320    2579 generic.go:356] "Generic (PLEG): container finished" podID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerID="5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb" exitCode=2
Jul 21 17:42:54.320426 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.320384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" event={"ID":"45b45d18-c05d-420d-9ef8-4da31c1fc48f","Type":"ContainerDied","Data":"5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb"}
Jul 21 17:42:54.327042 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.327016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz24n\" (UniqueName: \"kubernetes.io/projected/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-api-access-qz24n\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.471735 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.471710    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:42:54.519433 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.519408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-rbac-proxy-sar-config\") pod \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") "
Jul 21 17:42:54.519570 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.519480    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-proxy-tls\") pod \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") "
Jul 21 17:42:54.519570 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.519506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jlhn\" (UniqueName: \"kubernetes.io/projected/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-api-access-5jlhn\") pod \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\" (UID: \"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c\") "
Jul 21 17:42:54.519785 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.519761    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" (UID: "1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:42:54.521491 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.521466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-api-access-5jlhn" (OuterVolumeSpecName: "kube-api-access-5jlhn") pod "1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" (UID: "1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c"). InnerVolumeSpecName "kube-api-access-5jlhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:42:54.521588 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.521518    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" (UID: "1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:42:54.620135 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.620110    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:42:54.620135 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.620133    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:42:54.620259 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.620144    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jlhn\" (UniqueName: \"kubernetes.io/projected/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c-kube-api-access-5jlhn\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:42:54.821815 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.821741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b15d8fb-fd37-4092-a82d-855f29707a03-proxy-tls\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:54.824028 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:54.824009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b15d8fb-fd37-4092-a82d-855f29707a03-proxy-tls\") pod \"isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:55.093350 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.093271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:42:55.227976 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.227952    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"]
Jul 21 17:42:55.229705 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:42:55.229679    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b15d8fb_fd37_4092_a82d_855f29707a03.slice/crio-11cc048c85badf18d9438c33b377d253ef89969831a43c7197e419ac9ceffc1f WatchSource:0}: Error finding container 11cc048c85badf18d9438c33b377d253ef89969831a43c7197e419ac9ceffc1f: Status 404 returned error can't find the container with id 11cc048c85badf18d9438c33b377d253ef89969831a43c7197e419ac9ceffc1f
Jul 21 17:42:55.327775 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.327746    2579 generic.go:356] "Generic (PLEG): container finished" podID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" containerID="5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2" exitCode=2
Jul 21 17:42:55.327775 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.327774    2579 generic.go:356] "Generic (PLEG): container finished" podID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" containerID="0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36" exitCode=2
Jul 21 17:42:55.327970 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.327795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg" event={"ID":"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c","Type":"ContainerDied","Data":"5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2"}
Jul 21 17:42:55.327970 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.327808    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"
Jul 21 17:42:55.327970 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.327857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg" event={"ID":"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c","Type":"ContainerDied","Data":"0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36"}
Jul 21 17:42:55.327970 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.327887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg" event={"ID":"1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c","Type":"ContainerDied","Data":"797c42f261d02e496713e461ed251c7432503c2697cd1b558347059c58c6cd0f"}
Jul 21 17:42:55.327970 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.327911    2579 scope.go:117] "RemoveContainer" containerID="5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2"
Jul 21 17:42:55.329747 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.329647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" event={"ID":"7b15d8fb-fd37-4092-a82d-855f29707a03","Type":"ContainerStarted","Data":"26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165"}
Jul 21 17:42:55.329904 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.329863    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" event={"ID":"7b15d8fb-fd37-4092-a82d-855f29707a03","Type":"ContainerStarted","Data":"11cc048c85badf18d9438c33b377d253ef89969831a43c7197e419ac9ceffc1f"}
Jul 21 17:42:55.337116 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.337094    2579 scope.go:117] "RemoveContainer" containerID="0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36"
Jul 21 17:42:55.345403 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.345351    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"]
Jul 21 17:42:55.346776 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.346760    2579 scope.go:117] "RemoveContainer" containerID="5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2"
Jul 21 17:42:55.347345 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:42:55.347323    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2\": container with ID starting with 5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2 not found: ID does not exist" containerID="5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2"
Jul 21 17:42:55.347426 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.347356    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2"} err="failed to get container status \"5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2\": rpc error: code = NotFound desc = could not find container \"5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2\": container with ID starting with 5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2 not found: ID does not exist"
Jul 21 17:42:55.347426 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.347377    2579 scope.go:117] "RemoveContainer" containerID="0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36"
Jul 21 17:42:55.347625 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:42:55.347601    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36\": container with ID starting with 0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36 not found: ID does not exist" containerID="0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36"
Jul 21 17:42:55.347742 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.347626    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36"} err="failed to get container status \"0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36\": rpc error: code = NotFound desc = could not find container \"0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36\": container with ID starting with 0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36 not found: ID does not exist"
Jul 21 17:42:55.347742 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.347643    2579 scope.go:117] "RemoveContainer" containerID="5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2"
Jul 21 17:42:55.347920 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.347902    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2"} err="failed to get container status \"5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2\": rpc error: code = NotFound desc = could not find container \"5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2\": container with ID starting with 5f3b90b482bdc8c6e581ff575b1963ee0bee5a2ee0903a9f3552f3ce54c5f4f2 not found: ID does not exist"
Jul 21 17:42:55.347976 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.347921    2579 scope.go:117] "RemoveContainer" containerID="0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36"
Jul 21 17:42:55.348174 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.348158    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36"} err="failed to get container status \"0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36\": rpc error: code = NotFound desc = could not find container \"0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36\": container with ID starting with 0514000cd1c97bf090df5910d12d8b53580a9276ae9c164f799c06c27582dd36 not found: ID does not exist"
Jul 21 17:42:55.350681 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:55.350643    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4098a-predictor-f7bdb8d96-bkghg"]
Jul 21 17:42:56.932980 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:56.932941    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 21 17:42:57.004232 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:57.004202    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" path="/var/lib/kubelet/pods/1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c/volumes"
Jul 21 17:42:58.345908 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:58.345882    2579 generic.go:356] "Generic (PLEG): container finished" podID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerID="6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c" exitCode=0
Jul 21 17:42:58.346237 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:58.345954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" event={"ID":"45b45d18-c05d-420d-9ef8-4da31c1fc48f","Type":"ContainerDied","Data":"6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c"}
Jul 21 17:42:59.351947 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:59.351914    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerID="26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165" exitCode=0
Jul 21 17:42:59.352326 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:42:59.351985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" event={"ID":"7b15d8fb-fd37-4092-a82d-855f29707a03","Type":"ContainerDied","Data":"26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165"}
Jul 21 17:43:00.358740 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:00.358707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" event={"ID":"7b15d8fb-fd37-4092-a82d-855f29707a03","Type":"ContainerStarted","Data":"8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3"}
Jul 21 17:43:00.359100 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:00.358792    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" event={"ID":"7b15d8fb-fd37-4092-a82d-855f29707a03","Type":"ContainerStarted","Data":"402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c"}
Jul 21 17:43:00.359100 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:00.358977    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:43:00.424723 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:00.424671    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podStartSLOduration=2.882453718 podStartE2EDuration="6.424637882s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.542184164s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:43:00.423529729 +0000 UTC m=+1005.982328712" watchObservedRunningTime="2026-07-21 17:43:00.424637882 +0000 UTC m=+1005.983436866"
Jul 21 17:43:01.364351 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:01.364318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:43:01.365727 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:01.365702    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:43:01.932466 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:01.932427    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 21 17:43:01.937781 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:01.937751    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:43:01.938130 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:01.938109    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:43:02.369521 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:02.369485    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:43:06.933022 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:06.932979    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 21 17:43:06.933418 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:06.933142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:43:07.374135 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:07.374107    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:43:07.374611 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:07.374583    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:43:11.932898 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:11.932859    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 21 17:43:11.938235 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:11.938206    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:43:11.938614 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:11.938548    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:43:16.932954 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:16.932909    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 21 17:43:17.374765 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:17.374731    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:43:21.933028 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:21.932980    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 21 17:43:21.937372 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:21.937337    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:43:21.937485 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:21.937469    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:43:21.937708 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:21.937672    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:43:21.937816 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:21.937802    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:43:24.271892 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.271862    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:43:24.370265 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.370183    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b45d18-c05d-420d-9ef8-4da31c1fc48f-proxy-tls\") pod \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") "
Jul 21 17:43:24.370265 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.370226    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-cabundle-cert\") pod \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") "
Jul 21 17:43:24.370504 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.370276    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-rbac-proxy-sar-config\") pod \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") "
Jul 21 17:43:24.370504 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.370304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kserve-provision-location\") pod \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") "
Jul 21 17:43:24.370504 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.370326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pcfl\" (UniqueName: \"kubernetes.io/projected/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-api-access-6pcfl\") pod \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\" (UID: \"45b45d18-c05d-420d-9ef8-4da31c1fc48f\") "
Jul 21 17:43:24.370695 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.370619    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45b45d18-c05d-420d-9ef8-4da31c1fc48f" (UID: "45b45d18-c05d-420d-9ef8-4da31c1fc48f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:43:24.370695 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.370632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45b45d18-c05d-420d-9ef8-4da31c1fc48f" (UID: "45b45d18-c05d-420d-9ef8-4da31c1fc48f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:43:24.370695 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.370641    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45b45d18-c05d-420d-9ef8-4da31c1fc48f" (UID: "45b45d18-c05d-420d-9ef8-4da31c1fc48f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:43:24.372395 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.372370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b45d18-c05d-420d-9ef8-4da31c1fc48f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45b45d18-c05d-420d-9ef8-4da31c1fc48f" (UID: "45b45d18-c05d-420d-9ef8-4da31c1fc48f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:43:24.372487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.372423    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-api-access-6pcfl" (OuterVolumeSpecName: "kube-api-access-6pcfl") pod "45b45d18-c05d-420d-9ef8-4da31c1fc48f" (UID: "45b45d18-c05d-420d-9ef8-4da31c1fc48f"). InnerVolumeSpecName "kube-api-access-6pcfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:43:24.468182 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.468146    2579 generic.go:356] "Generic (PLEG): container finished" podID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerID="6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1" exitCode=0
Jul 21 17:43:24.468328 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.468215    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"
Jul 21 17:43:24.468328 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.468225    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" event={"ID":"45b45d18-c05d-420d-9ef8-4da31c1fc48f","Type":"ContainerDied","Data":"6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1"}
Jul 21 17:43:24.468328 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.468296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5" event={"ID":"45b45d18-c05d-420d-9ef8-4da31c1fc48f","Type":"ContainerDied","Data":"7f918d9d74796163d4e3352ed18f0989377ee6a9f0cc3a20959fb602ef72091f"}
Jul 21 17:43:24.468328 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.468327    2579 scope.go:117] "RemoveContainer" containerID="6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1"
Jul 21 17:43:24.471057 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.471037    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:43:24.471132 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.471065    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:43:24.471132 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.471080    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pcfl\" (UniqueName: \"kubernetes.io/projected/45b45d18-c05d-420d-9ef8-4da31c1fc48f-kube-api-access-6pcfl\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:43:24.471132 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.471096    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b45d18-c05d-420d-9ef8-4da31c1fc48f-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:43:24.471132 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.471112    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b45d18-c05d-420d-9ef8-4da31c1fc48f-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:43:24.476808 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.476792    2579 scope.go:117] "RemoveContainer" containerID="5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb"
Jul 21 17:43:24.486432 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.486414    2579 scope.go:117] "RemoveContainer" containerID="6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c"
Jul 21 17:43:24.490948 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.490928    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"]
Jul 21 17:43:24.493680 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.493644    2579 scope.go:117] "RemoveContainer" containerID="8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6"
Jul 21 17:43:24.496932 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.496910    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4098a-predictor-7d7bdfd689-288x5"]
Jul 21 17:43:24.501193 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.501176    2579 scope.go:117] "RemoveContainer" containerID="6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1"
Jul 21 17:43:24.501445 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:43:24.501429    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1\": container with ID starting with 6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1 not found: ID does not exist" containerID="6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1"
Jul 21 17:43:24.501492 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.501455    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1"} err="failed to get container status \"6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1\": rpc error: code = NotFound desc = could not find container \"6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1\": container with ID starting with 6c7a497a5e1a67c6c08e25a2c2d830d9904823ecc12afc59166d1820e15899c1 not found: ID does not exist"
Jul 21 17:43:24.501492 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.501472    2579 scope.go:117] "RemoveContainer" containerID="5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb"
Jul 21 17:43:24.501734 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:43:24.501717    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb\": container with ID starting with 5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb not found: ID does not exist" containerID="5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb"
Jul 21 17:43:24.501788 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.501742    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb"} err="failed to get container status \"5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb\": rpc error: code = NotFound desc = could not find container \"5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb\": container with ID starting with 5d880f8f61b116e39508bedbe1ccd45d8452c2ef0a0ffe1c0f1a7c9cda54c5cb not found: ID does not exist"
Jul 21 17:43:24.501788 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.501759    2579 scope.go:117] "RemoveContainer" containerID="6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c"
Jul 21 17:43:24.501976 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:43:24.501955    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c\": container with ID starting with 6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c not found: ID does not exist" containerID="6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c"
Jul 21 17:43:24.502055 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.501979    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c"} err="failed to get container status \"6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c\": rpc error: code = NotFound desc = could not find container \"6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c\": container with ID starting with 6574fa7f1072d389fb33f60c7684f084cdb96dc9bae13e3c9c69fdefb6a44a0c not found: ID does not exist"
Jul 21 17:43:24.502055 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.501994    2579 scope.go:117] "RemoveContainer" containerID="8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6"
Jul 21 17:43:24.502210 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:43:24.502194    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6\": container with ID starting with 8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6 not found: ID does not exist" containerID="8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6"
Jul 21 17:43:24.502256 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:24.502213    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6"} err="failed to get container status \"8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6\": rpc error: code = NotFound desc = could not find container \"8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6\": container with ID starting with 8342b14e8b8397d94654af3f1a0ee019d39ca6d448ec3ff56db392064bb893f6 not found: ID does not exist"
Jul 21 17:43:25.003810 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:25.003779    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" path="/var/lib/kubelet/pods/45b45d18-c05d-420d-9ef8-4da31c1fc48f/volumes"
Jul 21 17:43:27.374812 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:27.374767    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:43:37.374586 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:37.374545    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:43:47.375038 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:47.375002    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:43:57.375282 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:43:57.375245    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:44:07.375305 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:44:07.375217    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:44:17.375387 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:44:17.375342    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:44:18.999176 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:44:18.999132    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:44:29.000188 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:44:29.000136    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:44:38.999119 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:44:38.999067    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:44:49.000041 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:44:49.000003    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:44:58.999127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:44:58.999078    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:45:08.999087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:08.999048    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:45:18.999140 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:18.999091    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:45:29.003453 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:29.003424    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:45:34.346761 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:34.346719    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"]
Jul 21 17:45:34.347236 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:34.346974    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" containerID="cri-o://402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c" gracePeriod=30
Jul 21 17:45:34.347236 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:34.346975    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kube-rbac-proxy" containerID="cri-o://8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3" gracePeriod=30
Jul 21 17:45:35.018871 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:35.018838    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerID="8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3" exitCode=2
Jul 21 17:45:35.019131 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:35.018919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" event={"ID":"7b15d8fb-fd37-4092-a82d-855f29707a03","Type":"ContainerDied","Data":"8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3"}
Jul 21 17:45:37.370816 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:37.370714    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 21 17:45:39.000003 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:38.999958    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:45:39.502087 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502044    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"]
Jul 21 17:45:39.502416 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502403    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container"
Jul 21 17:45:39.502466 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502418    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container"
Jul 21 17:45:39.502466 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502431    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" containerName="kube-rbac-proxy"
Jul 21 17:45:39.502466 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502436    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" containerName="kube-rbac-proxy"
Jul 21 17:45:39.502466 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502444    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent"
Jul 21 17:45:39.502466 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502455    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent"
Jul 21 17:45:39.502466 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502465    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" containerName="kserve-container"
Jul 21 17:45:39.502673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502473    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" containerName="kserve-container"
Jul 21 17:45:39.502673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502488    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="storage-initializer"
Jul 21 17:45:39.502673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502494    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="storage-initializer"
Jul 21 17:45:39.502673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502511    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kube-rbac-proxy"
Jul 21 17:45:39.502673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502516    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kube-rbac-proxy"
Jul 21 17:45:39.502673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502573    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kube-rbac-proxy"
Jul 21 17:45:39.502673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502583    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="agent"
Jul 21 17:45:39.502673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502588    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" containerName="kube-rbac-proxy"
Jul 21 17:45:39.502673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502595    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b45d18-c05d-420d-9ef8-4da31c1fc48f" containerName="kserve-container"
Jul 21 17:45:39.502673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.502602    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f080ef3-9e5b-4f7f-b17d-e8ff69f06b4c" containerName="kserve-container"
Jul 21 17:45:39.509693 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.509637    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.512493 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.512466    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-4e9225-kube-rbac-proxy-sar-config\""
Jul 21 17:45:39.512677 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.512505    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-4e9225-predictor-serving-cert\""
Jul 21 17:45:39.516494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.516469    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"]
Jul 21 17:45:39.653156 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.653117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-cabundle-cert\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.653359 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.653177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snnh\" (UniqueName: \"kubernetes.io/projected/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-api-access-6snnh\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.653359 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.653340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-proxy-tls\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.653486 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.653375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kserve-provision-location\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.653486 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.653411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-rbac-proxy-sar-config\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.753888 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.753806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-rbac-proxy-sar-config\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.753888 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.753848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-cabundle-cert\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.753888 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.753875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snnh\" (UniqueName: \"kubernetes.io/projected/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-api-access-6snnh\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.754200 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.753947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-proxy-tls\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.754200 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.753975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kserve-provision-location\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.754360 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.754341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kserve-provision-location\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.754499 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.754477    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-rbac-proxy-sar-config\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.754623 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.754601    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-cabundle-cert\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.756423 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.756403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-proxy-tls\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.762685 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.762637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snnh\" (UniqueName: \"kubernetes.io/projected/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-api-access-6snnh\") pod \"isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") " pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.821681 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.821611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:39.960034 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:39.960010    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"]
Jul 21 17:45:39.962276 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:45:39.962250    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca915421_af3e_4be7_80d4_3ef5f19bdcdf.slice/crio-ff13374c1e90bb04eec9185980f2bd71fee8f96c47791e92c575d45953151211 WatchSource:0}: Error finding container ff13374c1e90bb04eec9185980f2bd71fee8f96c47791e92c575d45953151211: Status 404 returned error can't find the container with id ff13374c1e90bb04eec9185980f2bd71fee8f96c47791e92c575d45953151211
Jul 21 17:45:40.040610 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:40.040579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" event={"ID":"ca915421-af3e-4be7-80d4-3ef5f19bdcdf","Type":"ContainerStarted","Data":"5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf"}
Jul 21 17:45:40.041015 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:40.040639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" event={"ID":"ca915421-af3e-4be7-80d4-3ef5f19bdcdf","Type":"ContainerStarted","Data":"ff13374c1e90bb04eec9185980f2bd71fee8f96c47791e92c575d45953151211"}
Jul 21 17:45:42.370262 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:42.370219    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 21 17:45:44.060354 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.060328    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerID="5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf" exitCode=0
Jul 21 17:45:44.060622 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.060384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" event={"ID":"ca915421-af3e-4be7-80d4-3ef5f19bdcdf","Type":"ContainerDied","Data":"5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf"}
Jul 21 17:45:44.232793 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.232768    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:45:44.290473 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.290433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-rbac-proxy-sar-config\") pod \"7b15d8fb-fd37-4092-a82d-855f29707a03\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") "
Jul 21 17:45:44.290473 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.290479    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-cabundle-cert\") pod \"7b15d8fb-fd37-4092-a82d-855f29707a03\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") "
Jul 21 17:45:44.290763 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.290523    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b15d8fb-fd37-4092-a82d-855f29707a03-proxy-tls\") pod \"7b15d8fb-fd37-4092-a82d-855f29707a03\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") "
Jul 21 17:45:44.290763 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.290543    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz24n\" (UniqueName: \"kubernetes.io/projected/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-api-access-qz24n\") pod \"7b15d8fb-fd37-4092-a82d-855f29707a03\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") "
Jul 21 17:45:44.290763 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.290569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b15d8fb-fd37-4092-a82d-855f29707a03-kserve-provision-location\") pod \"7b15d8fb-fd37-4092-a82d-855f29707a03\" (UID: \"7b15d8fb-fd37-4092-a82d-855f29707a03\") "
Jul 21 17:45:44.291025 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.290981    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b15d8fb-fd37-4092-a82d-855f29707a03" (UID: "7b15d8fb-fd37-4092-a82d-855f29707a03"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:45:44.291106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.291033    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b15d8fb-fd37-4092-a82d-855f29707a03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b15d8fb-fd37-4092-a82d-855f29707a03" (UID: "7b15d8fb-fd37-4092-a82d-855f29707a03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:45:44.291106 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.291066    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b15d8fb-fd37-4092-a82d-855f29707a03" (UID: "7b15d8fb-fd37-4092-a82d-855f29707a03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:45:44.292716 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.292692    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b15d8fb-fd37-4092-a82d-855f29707a03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b15d8fb-fd37-4092-a82d-855f29707a03" (UID: "7b15d8fb-fd37-4092-a82d-855f29707a03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:45:44.292796 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.292732    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-api-access-qz24n" (OuterVolumeSpecName: "kube-api-access-qz24n") pod "7b15d8fb-fd37-4092-a82d-855f29707a03" (UID: "7b15d8fb-fd37-4092-a82d-855f29707a03"). InnerVolumeSpecName "kube-api-access-qz24n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:45:44.391769 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.391727    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:45:44.391769 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.391762    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b15d8fb-fd37-4092-a82d-855f29707a03-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:45:44.391769 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.391773    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b15d8fb-fd37-4092-a82d-855f29707a03-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:45:44.391769 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.391781    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz24n\" (UniqueName: \"kubernetes.io/projected/7b15d8fb-fd37-4092-a82d-855f29707a03-kube-api-access-qz24n\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:45:44.392057 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:44.391790    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b15d8fb-fd37-4092-a82d-855f29707a03-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:45:45.072193 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.072088    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerID="402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c" exitCode=0
Jul 21 17:45:45.072193 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.072159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" event={"ID":"7b15d8fb-fd37-4092-a82d-855f29707a03","Type":"ContainerDied","Data":"402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c"}
Jul 21 17:45:45.072725 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.072172    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"
Jul 21 17:45:45.072725 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.072264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb" event={"ID":"7b15d8fb-fd37-4092-a82d-855f29707a03","Type":"ContainerDied","Data":"11cc048c85badf18d9438c33b377d253ef89969831a43c7197e419ac9ceffc1f"}
Jul 21 17:45:45.072725 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.072285    2579 scope.go:117] "RemoveContainer" containerID="8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3"
Jul 21 17:45:45.074942 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.074915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" event={"ID":"ca915421-af3e-4be7-80d4-3ef5f19bdcdf","Type":"ContainerStarted","Data":"b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03"}
Jul 21 17:45:45.075058 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.074993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" event={"ID":"ca915421-af3e-4be7-80d4-3ef5f19bdcdf","Type":"ContainerStarted","Data":"69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0"}
Jul 21 17:45:45.075466 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.075444    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:45.075558 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.075491    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:45.077157 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.077133    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:45:45.081048 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.081028    2579 scope.go:117] "RemoveContainer" containerID="402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c"
Jul 21 17:45:45.091058 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.091032    2579 scope.go:117] "RemoveContainer" containerID="26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165"
Jul 21 17:45:45.099254 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.099236    2579 scope.go:117] "RemoveContainer" containerID="8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3"
Jul 21 17:45:45.099430 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.099393    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podStartSLOduration=2.311636967 podStartE2EDuration="6.099381069s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.787744102s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:45:45.097614303 +0000 UTC m=+1170.656413288" watchObservedRunningTime="2026-07-21 17:45:45.099381069 +0000 UTC m=+1170.658180053"
Jul 21 17:45:45.099540 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:45:45.099519    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3\": container with ID starting with 8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3 not found: ID does not exist" containerID="8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3"
Jul 21 17:45:45.099583 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.099549    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3"} err="failed to get container status \"8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3\": rpc error: code = NotFound desc = could not find container \"8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3\": container with ID starting with 8a961bb85f06fb27b3a818c19736facee8bfa86542384438008f1f15860eb8e3 not found: ID does not exist"
Jul 21 17:45:45.099583 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.099568    2579 scope.go:117] "RemoveContainer" containerID="402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c"
Jul 21 17:45:45.099818 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:45:45.099800    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c\": container with ID starting with 402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c not found: ID does not exist" containerID="402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c"
Jul 21 17:45:45.099891 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.099826    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c"} err="failed to get container status \"402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c\": rpc error: code = NotFound desc = could not find container \"402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c\": container with ID starting with 402ad4bb50ba268aea0e4834c530bd219eaf10c8d5c816c4fbb2a86d7e74b18c not found: ID does not exist"
Jul 21 17:45:45.099891 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.099848    2579 scope.go:117] "RemoveContainer" containerID="26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165"
Jul 21 17:45:45.100126 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:45:45.100079    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165\": container with ID starting with 26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165 not found: ID does not exist" containerID="26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165"
Jul 21 17:45:45.100219 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.100130    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165"} err="failed to get container status \"26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165\": rpc error: code = NotFound desc = could not find container \"26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165\": container with ID starting with 26f8bcb7cff46ea5ed0460d84e6287a12ebe54ccbafde793af6393aae420e165 not found: ID does not exist"
Jul 21 17:45:45.111544 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.111521    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"]
Jul 21 17:45:45.114340 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:45.114320    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-90cae-predictor-75cc6f54c4-mfqfb"]
Jul 21 17:45:46.081127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:46.081094    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:45:47.003556 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:47.003521    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" path="/var/lib/kubelet/pods/7b15d8fb-fd37-4092-a82d-855f29707a03/volumes"
Jul 21 17:45:51.085529 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:51.085499    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:45:51.086090 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:45:51.086064    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:46:01.086127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:46:01.086084    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:46:11.086476 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:46:11.086439    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:46:21.086188 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:46:21.086145    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:46:31.086114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:46:31.086064    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:46:41.086624 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:46:41.086585    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:46:51.086389 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:46:51.086353    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:47:01.086819 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:01.086790    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:47:24.716012 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.715976    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"]
Jul 21 17:47:24.716484 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.716306    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="storage-initializer"
Jul 21 17:47:24.716484 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.716316    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="storage-initializer"
Jul 21 17:47:24.716484 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.716328    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kube-rbac-proxy"
Jul 21 17:47:24.716484 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.716334    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kube-rbac-proxy"
Jul 21 17:47:24.716484 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.716345    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container"
Jul 21 17:47:24.716484 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.716351    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container"
Jul 21 17:47:24.716484 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.716406    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kserve-container"
Jul 21 17:47:24.716484 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.716414    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b15d8fb-fd37-4092-a82d-855f29707a03" containerName="kube-rbac-proxy"
Jul 21 17:47:24.719627 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.719610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.722686 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.722667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-4e9225\""
Jul 21 17:47:24.724179 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.724155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-4e9225-predictor-serving-cert\""
Jul 21 17:47:24.724179 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.724179    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-4e9225-dockercfg-wjckf\""
Jul 21 17:47:24.724307 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.724209    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-4e9225-kube-rbac-proxy-sar-config\""
Jul 21 17:47:24.730552 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.730531    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"]
Jul 21 17:47:24.794240 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.794210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-proxy-tls\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.794359 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.794252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-cabundle-cert\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.794359 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.794276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pb5\" (UniqueName: \"kubernetes.io/projected/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-api-access-x5pb5\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.794359 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.794336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kserve-provision-location\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.794471 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.794382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.895300 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.895260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-proxy-tls\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.895450 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.895311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-cabundle-cert\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.895450 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.895428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pb5\" (UniqueName: \"kubernetes.io/projected/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-api-access-x5pb5\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.895524 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.895460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kserve-provision-location\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.895568 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.895531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.895934 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.895902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kserve-provision-location\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.896074 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.895937    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-cabundle-cert\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.896221 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.896202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.897722 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.897704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-proxy-tls\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:24.904257 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:24.904235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pb5\" (UniqueName: \"kubernetes.io/projected/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-api-access-x5pb5\") pod \"isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") " pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:25.030164 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:25.030100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:25.164744 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:25.164718    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"]
Jul 21 17:47:25.167185 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:47:25.167159    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b0984c_5c5f_453d_8f6a_018d0e6e3d02.slice/crio-33fd46488437a6cd03aa9963f72a956327cded8d2985f828048dbd1dbf3f12b2 WatchSource:0}: Error finding container 33fd46488437a6cd03aa9963f72a956327cded8d2985f828048dbd1dbf3f12b2: Status 404 returned error can't find the container with id 33fd46488437a6cd03aa9963f72a956327cded8d2985f828048dbd1dbf3f12b2
Jul 21 17:47:25.494871 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:25.494833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5" event={"ID":"07b0984c-5c5f-453d-8f6a-018d0e6e3d02","Type":"ContainerStarted","Data":"85ee7f28eee04a9e1dd663cadc2c059258079febe170cc20dad9577a5031beac"}
Jul 21 17:47:25.495102 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:25.494891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5" event={"ID":"07b0984c-5c5f-453d-8f6a-018d0e6e3d02","Type":"ContainerStarted","Data":"33fd46488437a6cd03aa9963f72a956327cded8d2985f828048dbd1dbf3f12b2"}
Jul 21 17:47:29.516594 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:29.516513    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5_07b0984c-5c5f-453d-8f6a-018d0e6e3d02/storage-initializer/0.log"
Jul 21 17:47:29.516594 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:29.516552    2579 generic.go:356] "Generic (PLEG): container finished" podID="07b0984c-5c5f-453d-8f6a-018d0e6e3d02" containerID="85ee7f28eee04a9e1dd663cadc2c059258079febe170cc20dad9577a5031beac" exitCode=1
Jul 21 17:47:29.517128 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:29.516645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5" event={"ID":"07b0984c-5c5f-453d-8f6a-018d0e6e3d02","Type":"ContainerDied","Data":"85ee7f28eee04a9e1dd663cadc2c059258079febe170cc20dad9577a5031beac"}
Jul 21 17:47:30.522971 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:30.522946    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5_07b0984c-5c5f-453d-8f6a-018d0e6e3d02/storage-initializer/0.log"
Jul 21 17:47:30.523428 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:30.523040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5" event={"ID":"07b0984c-5c5f-453d-8f6a-018d0e6e3d02","Type":"ContainerStarted","Data":"a027efa5ee130315f41e8e6219e3b9aa84b343f4df33b74463fc87c77ca42554"}
Jul 21 17:47:34.542254 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:34.542227    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5_07b0984c-5c5f-453d-8f6a-018d0e6e3d02/storage-initializer/1.log"
Jul 21 17:47:34.542640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:34.542580    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5_07b0984c-5c5f-453d-8f6a-018d0e6e3d02/storage-initializer/0.log"
Jul 21 17:47:34.542640 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:34.542612    2579 generic.go:356] "Generic (PLEG): container finished" podID="07b0984c-5c5f-453d-8f6a-018d0e6e3d02" containerID="a027efa5ee130315f41e8e6219e3b9aa84b343f4df33b74463fc87c77ca42554" exitCode=1
Jul 21 17:47:34.542736 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:34.542694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5" event={"ID":"07b0984c-5c5f-453d-8f6a-018d0e6e3d02","Type":"ContainerDied","Data":"a027efa5ee130315f41e8e6219e3b9aa84b343f4df33b74463fc87c77ca42554"}
Jul 21 17:47:34.542772 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:34.542756    2579 scope.go:117] "RemoveContainer" containerID="85ee7f28eee04a9e1dd663cadc2c059258079febe170cc20dad9577a5031beac"
Jul 21 17:47:34.543163 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:34.543144    2579 scope.go:117] "RemoveContainer" containerID="85ee7f28eee04a9e1dd663cadc2c059258079febe170cc20dad9577a5031beac"
Jul 21 17:47:34.553629 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:47:34.553609    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5_kserve-ci-e2e-test(07b0984c-5c5f-453d-8f6a-018d0e6e3d02)\"" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5" podUID="07b0984c-5c5f-453d-8f6a-018d0e6e3d02"
Jul 21 17:47:34.553997 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:47:34.553968    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5_kserve-ci-e2e-test_07b0984c-5c5f-453d-8f6a-018d0e6e3d02_0 in pod sandbox 33fd46488437a6cd03aa9963f72a956327cded8d2985f828048dbd1dbf3f12b2 from index: no such id: '85ee7f28eee04a9e1dd663cadc2c059258079febe170cc20dad9577a5031beac'" containerID="85ee7f28eee04a9e1dd663cadc2c059258079febe170cc20dad9577a5031beac"
Jul 21 17:47:34.554103 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:34.554005    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ee7f28eee04a9e1dd663cadc2c059258079febe170cc20dad9577a5031beac"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5_kserve-ci-e2e-test_07b0984c-5c5f-453d-8f6a-018d0e6e3d02_0 in pod sandbox 33fd46488437a6cd03aa9963f72a956327cded8d2985f828048dbd1dbf3f12b2 from index: no such id: '85ee7f28eee04a9e1dd663cadc2c059258079febe170cc20dad9577a5031beac'"
Jul 21 17:47:35.548435 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:35.548406    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5_07b0984c-5c5f-453d-8f6a-018d0e6e3d02/storage-initializer/1.log"
Jul 21 17:47:40.767921 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:40.767879    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"]
Jul 21 17:47:40.827305 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:40.827275    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"]
Jul 21 17:47:40.827605 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:40.827578    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" containerID="cri-o://69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0" gracePeriod=30
Jul 21 17:47:40.827707 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:40.827604    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kube-rbac-proxy" containerID="cri-o://b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03" gracePeriod=30
Jul 21 17:47:40.947487 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:40.947467    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5_07b0984c-5c5f-453d-8f6a-018d0e6e3d02/storage-initializer/1.log"
Jul 21 17:47:40.949606 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:40.949592    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:41.039529 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.039467    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-rbac-proxy-sar-config\") pod \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") "
Jul 21 17:47:41.039529 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.039500    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-cabundle-cert\") pod \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") "
Jul 21 17:47:41.039529 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.039521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-proxy-tls\") pod \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") "
Jul 21 17:47:41.039927 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.039553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5pb5\" (UniqueName: \"kubernetes.io/projected/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-api-access-x5pb5\") pod \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") "
Jul 21 17:47:41.039927 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.039585    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kserve-provision-location\") pod \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\" (UID: \"07b0984c-5c5f-453d-8f6a-018d0e6e3d02\") "
Jul 21 17:47:41.039927 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.039888    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "07b0984c-5c5f-453d-8f6a-018d0e6e3d02" (UID: "07b0984c-5c5f-453d-8f6a-018d0e6e3d02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:47:41.039927 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.039898    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "07b0984c-5c5f-453d-8f6a-018d0e6e3d02" (UID: "07b0984c-5c5f-453d-8f6a-018d0e6e3d02"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:47:41.039927 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.039912    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07b0984c-5c5f-453d-8f6a-018d0e6e3d02" (UID: "07b0984c-5c5f-453d-8f6a-018d0e6e3d02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:47:41.041573 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.041548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07b0984c-5c5f-453d-8f6a-018d0e6e3d02" (UID: "07b0984c-5c5f-453d-8f6a-018d0e6e3d02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:47:41.041701 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.041591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-api-access-x5pb5" (OuterVolumeSpecName: "kube-api-access-x5pb5") pod "07b0984c-5c5f-453d-8f6a-018d0e6e3d02" (UID: "07b0984c-5c5f-453d-8f6a-018d0e6e3d02"). InnerVolumeSpecName "kube-api-access-x5pb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:47:41.082125 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.082095    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 21 17:47:41.086389 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.086368    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:47:41.140989 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.140962    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:41.140989 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.140988    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:41.141135 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.140998    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:41.141135 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.141006    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5pb5\" (UniqueName: \"kubernetes.io/projected/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kube-api-access-x5pb5\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:41.141135 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.141015    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07b0984c-5c5f-453d-8f6a-018d0e6e3d02-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:41.574884 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.574861    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5_07b0984c-5c5f-453d-8f6a-018d0e6e3d02/storage-initializer/1.log"
Jul 21 17:47:41.575057 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.574954    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"
Jul 21 17:47:41.575057 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.574956    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5" event={"ID":"07b0984c-5c5f-453d-8f6a-018d0e6e3d02","Type":"ContainerDied","Data":"33fd46488437a6cd03aa9963f72a956327cded8d2985f828048dbd1dbf3f12b2"}
Jul 21 17:47:41.575057 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.575024    2579 scope.go:117] "RemoveContainer" containerID="a027efa5ee130315f41e8e6219e3b9aa84b343f4df33b74463fc87c77ca42554"
Jul 21 17:47:41.577330 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.577306    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerID="b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03" exitCode=2
Jul 21 17:47:41.577449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.577378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" event={"ID":"ca915421-af3e-4be7-80d4-3ef5f19bdcdf","Type":"ContainerDied","Data":"b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03"}
Jul 21 17:47:41.613626 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.613603    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"]
Jul 21 17:47:41.614958 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:41.614936    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e9225-predictor-7c68b4f8-5nwt5"]
Jul 21 17:47:43.003127 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:43.003094    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07b0984c-5c5f-453d-8f6a-018d0e6e3d02" path="/var/lib/kubelet/pods/07b0984c-5c5f-453d-8f6a-018d0e6e3d02/volumes"
Jul 21 17:47:45.375964 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.375936    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:47:45.475820 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.475725    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-rbac-proxy-sar-config\") pod \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") "
Jul 21 17:47:45.475820 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.475773    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kserve-provision-location\") pod \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") "
Jul 21 17:47:45.476050 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.475827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-proxy-tls\") pod \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") "
Jul 21 17:47:45.476050 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.475879    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6snnh\" (UniqueName: \"kubernetes.io/projected/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-api-access-6snnh\") pod \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") "
Jul 21 17:47:45.476050 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.475921    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-cabundle-cert\") pod \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\" (UID: \"ca915421-af3e-4be7-80d4-3ef5f19bdcdf\") "
Jul 21 17:47:45.476192 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.476151    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca915421-af3e-4be7-80d4-3ef5f19bdcdf" (UID: "ca915421-af3e-4be7-80d4-3ef5f19bdcdf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:47:45.476192 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.476181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca915421-af3e-4be7-80d4-3ef5f19bdcdf" (UID: "ca915421-af3e-4be7-80d4-3ef5f19bdcdf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:47:45.476362 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.476340    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca915421-af3e-4be7-80d4-3ef5f19bdcdf" (UID: "ca915421-af3e-4be7-80d4-3ef5f19bdcdf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:47:45.478041 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.478015    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca915421-af3e-4be7-80d4-3ef5f19bdcdf" (UID: "ca915421-af3e-4be7-80d4-3ef5f19bdcdf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:47:45.478096 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.478051    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-api-access-6snnh" (OuterVolumeSpecName: "kube-api-access-6snnh") pod "ca915421-af3e-4be7-80d4-3ef5f19bdcdf" (UID: "ca915421-af3e-4be7-80d4-3ef5f19bdcdf"). InnerVolumeSpecName "kube-api-access-6snnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:47:45.577463 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.577429    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:45.577463 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.577457    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:45.577463 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.577467    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:45.577727 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.577476    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:45.577727 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.577486    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6snnh\" (UniqueName: \"kubernetes.io/projected/ca915421-af3e-4be7-80d4-3ef5f19bdcdf-kube-api-access-6snnh\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:45.597012 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.596982    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerID="69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0" exitCode=0
Jul 21 17:47:45.597132 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.597050    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"
Jul 21 17:47:45.597183 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.597052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" event={"ID":"ca915421-af3e-4be7-80d4-3ef5f19bdcdf","Type":"ContainerDied","Data":"69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0"}
Jul 21 17:47:45.597230 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.597217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5" event={"ID":"ca915421-af3e-4be7-80d4-3ef5f19bdcdf","Type":"ContainerDied","Data":"ff13374c1e90bb04eec9185980f2bd71fee8f96c47791e92c575d45953151211"}
Jul 21 17:47:45.597267 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.597254    2579 scope.go:117] "RemoveContainer" containerID="b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03"
Jul 21 17:47:45.606064 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.606043    2579 scope.go:117] "RemoveContainer" containerID="69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0"
Jul 21 17:47:45.613494 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.613481    2579 scope.go:117] "RemoveContainer" containerID="5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf"
Jul 21 17:47:45.621771 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.621751    2579 scope.go:117] "RemoveContainer" containerID="b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03"
Jul 21 17:47:45.622071 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:47:45.622024    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03\": container with ID starting with b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03 not found: ID does not exist" containerID="b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03"
Jul 21 17:47:45.622071 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.622060    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"]
Jul 21 17:47:45.622071 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.622060    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03"} err="failed to get container status \"b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03\": rpc error: code = NotFound desc = could not find container \"b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03\": container with ID starting with b81a2fc64fbd70ed91d7c1db6759a70a334ac1db57998d696898944022838a03 not found: ID does not exist"
Jul 21 17:47:45.622330 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.622083    2579 scope.go:117] "RemoveContainer" containerID="69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0"
Jul 21 17:47:45.622443 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:47:45.622415    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0\": container with ID starting with 69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0 not found: ID does not exist" containerID="69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0"
Jul 21 17:47:45.622601 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.622451    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0"} err="failed to get container status \"69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0\": rpc error: code = NotFound desc = could not find container \"69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0\": container with ID starting with 69a8738511ff9cb7d82d598cbec6fc3908f22f670d317f7e1056ec1ef91323b0 not found: ID does not exist"
Jul 21 17:47:45.622601 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.622474    2579 scope.go:117] "RemoveContainer" containerID="5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf"
Jul 21 17:47:45.622902 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:47:45.622755    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf\": container with ID starting with 5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf not found: ID does not exist" containerID="5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf"
Jul 21 17:47:45.622902 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.622781    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf"} err="failed to get container status \"5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf\": rpc error: code = NotFound desc = could not find container \"5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf\": container with ID starting with 5c8436d2958236df606297a430f81a53c1305ed1da98f405c344eceeed3551cf not found: ID does not exist"
Jul 21 17:47:45.624857 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.624839    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e9225-predictor-65bcb5bcfc-ngrn5"]
Jul 21 17:47:45.991016 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.990979    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"]
Jul 21 17:47:45.991304 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991292    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="storage-initializer"
Jul 21 17:47:45.991304 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991305    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="storage-initializer"
Jul 21 17:47:45.991420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991319    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kube-rbac-proxy"
Jul 21 17:47:45.991420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991327    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kube-rbac-proxy"
Jul 21 17:47:45.991420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991344    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b0984c-5c5f-453d-8f6a-018d0e6e3d02" containerName="storage-initializer"
Jul 21 17:47:45.991420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991353    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b0984c-5c5f-453d-8f6a-018d0e6e3d02" containerName="storage-initializer"
Jul 21 17:47:45.991420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991362    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b0984c-5c5f-453d-8f6a-018d0e6e3d02" containerName="storage-initializer"
Jul 21 17:47:45.991420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991369    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b0984c-5c5f-453d-8f6a-018d0e6e3d02" containerName="storage-initializer"
Jul 21 17:47:45.991420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991377    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container"
Jul 21 17:47:45.991420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991383    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container"
Jul 21 17:47:45.991695 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991453    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b0984c-5c5f-453d-8f6a-018d0e6e3d02" containerName="storage-initializer"
Jul 21 17:47:45.991695 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991462    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kserve-container"
Jul 21 17:47:45.991695 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991470    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" containerName="kube-rbac-proxy"
Jul 21 17:47:45.991695 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.991574    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b0984c-5c5f-453d-8f6a-018d0e6e3d02" containerName="storage-initializer"
Jul 21 17:47:45.995954 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.995934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:45.999114 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.999091    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:47:45.999224 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.999098    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-d6e639\""
Jul 21 17:47:45.999224 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.999126    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-d6e639-dockercfg-4kgdv\""
Jul 21 17:47:45.999224 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.999098    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-d6e639-predictor-serving-cert\""
Jul 21 17:47:45.999389 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.999256    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:47:45.999389 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.999290    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-d6e639-kube-rbac-proxy-sar-config\""
Jul 21 17:47:45.999547 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:45.999534    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:47:46.006214 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.006191    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"]
Jul 21 17:47:46.082210 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.082174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-cabundle-cert\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.082210 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.082211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kserve-provision-location\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.082420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.082234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsrqp\" (UniqueName: \"kubernetes.io/projected/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-api-access-xsrqp\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.082420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.082331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.082420 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.082392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1363ccb-7c98-4c7f-bf24-4052e27489eb-proxy-tls\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.183497 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.183467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-cabundle-cert\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.183700 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.183511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kserve-provision-location\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.183700 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.183539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrqp\" (UniqueName: \"kubernetes.io/projected/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-api-access-xsrqp\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.183700 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.183575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.183700 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.183644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1363ccb-7c98-4c7f-bf24-4052e27489eb-proxy-tls\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.183981 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.183956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kserve-provision-location\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.184236 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.184220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.184284 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.184238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-cabundle-cert\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.186035 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.186014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1363ccb-7c98-4c7f-bf24-4052e27489eb-proxy-tls\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.193151 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.193132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrqp\" (UniqueName: \"kubernetes.io/projected/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-api-access-xsrqp\") pod \"isvc-init-fail-d6e639-predictor-5449f794b-jlrqj\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.306725 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.306611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:46.455776 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.455748    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"]
Jul 21 17:47:46.456971 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:47:46.456944    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1363ccb_7c98_4c7f_bf24_4052e27489eb.slice/crio-2fb72ffa831d63376c305b9968e8e82ed63f64ef549a958556ce49f33ab6710a WatchSource:0}: Error finding container 2fb72ffa831d63376c305b9968e8e82ed63f64ef549a958556ce49f33ab6710a: Status 404 returned error can't find the container with id 2fb72ffa831d63376c305b9968e8e82ed63f64ef549a958556ce49f33ab6710a
Jul 21 17:47:46.603838 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.603801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj" event={"ID":"f1363ccb-7c98-4c7f-bf24-4052e27489eb","Type":"ContainerStarted","Data":"24cb7f152c9b99317f2677647154d3f10ab3f7f0cbc6ac6b7f14a8c6ebc6ffc1"}
Jul 21 17:47:46.604016 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:46.603877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj" event={"ID":"f1363ccb-7c98-4c7f-bf24-4052e27489eb","Type":"ContainerStarted","Data":"2fb72ffa831d63376c305b9968e8e82ed63f64ef549a958556ce49f33ab6710a"}
Jul 21 17:47:47.004042 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:47.003957    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca915421-af3e-4be7-80d4-3ef5f19bdcdf" path="/var/lib/kubelet/pods/ca915421-af3e-4be7-80d4-3ef5f19bdcdf/volumes"
Jul 21 17:47:51.627153 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:51.627126    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d6e639-predictor-5449f794b-jlrqj_f1363ccb-7c98-4c7f-bf24-4052e27489eb/storage-initializer/0.log"
Jul 21 17:47:51.627623 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:51.627165    2579 generic.go:356] "Generic (PLEG): container finished" podID="f1363ccb-7c98-4c7f-bf24-4052e27489eb" containerID="24cb7f152c9b99317f2677647154d3f10ab3f7f0cbc6ac6b7f14a8c6ebc6ffc1" exitCode=1
Jul 21 17:47:51.627623 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:51.627257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj" event={"ID":"f1363ccb-7c98-4c7f-bf24-4052e27489eb","Type":"ContainerDied","Data":"24cb7f152c9b99317f2677647154d3f10ab3f7f0cbc6ac6b7f14a8c6ebc6ffc1"}
Jul 21 17:47:52.634717 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:52.634677    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d6e639-predictor-5449f794b-jlrqj_f1363ccb-7c98-4c7f-bf24-4052e27489eb/storage-initializer/0.log"
Jul 21 17:47:52.635153 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:52.634754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj" event={"ID":"f1363ccb-7c98-4c7f-bf24-4052e27489eb","Type":"ContainerStarted","Data":"27cc104bb003e59105661190111942ff6de3e6cc4f5e897f6589e65949d00380"}
Jul 21 17:47:54.644505 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:54.644478    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d6e639-predictor-5449f794b-jlrqj_f1363ccb-7c98-4c7f-bf24-4052e27489eb/storage-initializer/1.log"
Jul 21 17:47:54.644940 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:54.644836    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d6e639-predictor-5449f794b-jlrqj_f1363ccb-7c98-4c7f-bf24-4052e27489eb/storage-initializer/0.log"
Jul 21 17:47:54.644940 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:54.644864    2579 generic.go:356] "Generic (PLEG): container finished" podID="f1363ccb-7c98-4c7f-bf24-4052e27489eb" containerID="27cc104bb003e59105661190111942ff6de3e6cc4f5e897f6589e65949d00380" exitCode=1
Jul 21 17:47:54.644940 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:54.644924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj" event={"ID":"f1363ccb-7c98-4c7f-bf24-4052e27489eb","Type":"ContainerDied","Data":"27cc104bb003e59105661190111942ff6de3e6cc4f5e897f6589e65949d00380"}
Jul 21 17:47:54.645090 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:54.644976    2579 scope.go:117] "RemoveContainer" containerID="24cb7f152c9b99317f2677647154d3f10ab3f7f0cbc6ac6b7f14a8c6ebc6ffc1"
Jul 21 17:47:54.645374 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:54.645349    2579 scope.go:117] "RemoveContainer" containerID="24cb7f152c9b99317f2677647154d3f10ab3f7f0cbc6ac6b7f14a8c6ebc6ffc1"
Jul 21 17:47:54.655477 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:47:54.655456    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-d6e639-predictor-5449f794b-jlrqj_kserve-ci-e2e-test(f1363ccb-7c98-4c7f-bf24-4052e27489eb)\"" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj" podUID="f1363ccb-7c98-4c7f-bf24-4052e27489eb"
Jul 21 17:47:54.655760 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:47:54.655733    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-d6e639-predictor-5449f794b-jlrqj_kserve-ci-e2e-test_f1363ccb-7c98-4c7f-bf24-4052e27489eb_0 in pod sandbox 2fb72ffa831d63376c305b9968e8e82ed63f64ef549a958556ce49f33ab6710a from index: no such id: '24cb7f152c9b99317f2677647154d3f10ab3f7f0cbc6ac6b7f14a8c6ebc6ffc1'" containerID="24cb7f152c9b99317f2677647154d3f10ab3f7f0cbc6ac6b7f14a8c6ebc6ffc1"
Jul 21 17:47:54.655813 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:54.655767    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24cb7f152c9b99317f2677647154d3f10ab3f7f0cbc6ac6b7f14a8c6ebc6ffc1"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-d6e639-predictor-5449f794b-jlrqj_kserve-ci-e2e-test_f1363ccb-7c98-4c7f-bf24-4052e27489eb_0 in pod sandbox 2fb72ffa831d63376c305b9968e8e82ed63f64ef549a958556ce49f33ab6710a from index: no such id: '24cb7f152c9b99317f2677647154d3f10ab3f7f0cbc6ac6b7f14a8c6ebc6ffc1'"
Jul 21 17:47:55.650833 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:55.650804    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d6e639-predictor-5449f794b-jlrqj_f1363ccb-7c98-4c7f-bf24-4052e27489eb/storage-initializer/1.log"
Jul 21 17:47:56.019104 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.019025    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"]
Jul 21 17:47:56.147793 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.147767    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d6e639-predictor-5449f794b-jlrqj_f1363ccb-7c98-4c7f-bf24-4052e27489eb/storage-initializer/1.log"
Jul 21 17:47:56.150034 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.150019    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:56.167116 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.167090    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kserve-provision-location\") pod \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") "
Jul 21 17:47:56.167216 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.167131    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsrqp\" (UniqueName: \"kubernetes.io/projected/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-api-access-xsrqp\") pod \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") "
Jul 21 17:47:56.167216 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.167176    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-cabundle-cert\") pod \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") "
Jul 21 17:47:56.167216 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.167194    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1363ccb-7c98-4c7f-bf24-4052e27489eb-proxy-tls\") pod \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") "
Jul 21 17:47:56.167216 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.167212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-rbac-proxy-sar-config\") pod \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\" (UID: \"f1363ccb-7c98-4c7f-bf24-4052e27489eb\") "
Jul 21 17:47:56.167430 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.167346    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1363ccb-7c98-4c7f-bf24-4052e27489eb" (UID: "f1363ccb-7c98-4c7f-bf24-4052e27489eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:47:56.167499 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.167475    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1363ccb-7c98-4c7f-bf24-4052e27489eb" (UID: "f1363ccb-7c98-4c7f-bf24-4052e27489eb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:47:56.167710 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.167684    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f1363ccb-7c98-4c7f-bf24-4052e27489eb" (UID: "f1363ccb-7c98-4c7f-bf24-4052e27489eb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:47:56.169215 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.169193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-api-access-xsrqp" (OuterVolumeSpecName: "kube-api-access-xsrqp") pod "f1363ccb-7c98-4c7f-bf24-4052e27489eb" (UID: "f1363ccb-7c98-4c7f-bf24-4052e27489eb"). InnerVolumeSpecName "kube-api-access-xsrqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:47:56.169215 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.169199    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1363ccb-7c98-4c7f-bf24-4052e27489eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1363ccb-7c98-4c7f-bf24-4052e27489eb" (UID: "f1363ccb-7c98-4c7f-bf24-4052e27489eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:47:56.268195 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.268170    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsrqp\" (UniqueName: \"kubernetes.io/projected/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-api-access-xsrqp\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:56.268195 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.268193    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:56.268351 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.268203    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1363ccb-7c98-4c7f-bf24-4052e27489eb-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:56.268351 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.268212    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:56.268351 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.268220    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1363ccb-7c98-4c7f-bf24-4052e27489eb-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:47:56.656208 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.656179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d6e639-predictor-5449f794b-jlrqj_f1363ccb-7c98-4c7f-bf24-4052e27489eb/storage-initializer/1.log"
Jul 21 17:47:56.656592 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.656257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj" event={"ID":"f1363ccb-7c98-4c7f-bf24-4052e27489eb","Type":"ContainerDied","Data":"2fb72ffa831d63376c305b9968e8e82ed63f64ef549a958556ce49f33ab6710a"}
Jul 21 17:47:56.656592 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.656288    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"
Jul 21 17:47:56.656592 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.656337    2579 scope.go:117] "RemoveContainer" containerID="27cc104bb003e59105661190111942ff6de3e6cc4f5e897f6589e65949d00380"
Jul 21 17:47:56.694444 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.694421    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"]
Jul 21 17:47:56.697844 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:56.697822    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d6e639-predictor-5449f794b-jlrqj"]
Jul 21 17:47:57.003219 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:47:57.003136    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1363ccb-7c98-4c7f-bf24-4052e27489eb" path="/var/lib/kubelet/pods/f1363ccb-7c98-4c7f-bf24-4052e27489eb/volumes"
Jul 21 17:48:01.209818 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.209789    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"]
Jul 21 17:48:01.210175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.210122    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1363ccb-7c98-4c7f-bf24-4052e27489eb" containerName="storage-initializer"
Jul 21 17:48:01.210175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.210134    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1363ccb-7c98-4c7f-bf24-4052e27489eb" containerName="storage-initializer"
Jul 21 17:48:01.210175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.210154    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1363ccb-7c98-4c7f-bf24-4052e27489eb" containerName="storage-initializer"
Jul 21 17:48:01.210175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.210161    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1363ccb-7c98-4c7f-bf24-4052e27489eb" containerName="storage-initializer"
Jul 21 17:48:01.210311 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.210209    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1363ccb-7c98-4c7f-bf24-4052e27489eb" containerName="storage-initializer"
Jul 21 17:48:01.210311 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.210219    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1363ccb-7c98-4c7f-bf24-4052e27489eb" containerName="storage-initializer"
Jul 21 17:48:01.215034 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.215013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.219050 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.218959    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-9487a-kube-rbac-proxy-sar-config\""
Jul 21 17:48:01.219050 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.218993    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:48:01.219050 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.219011    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:48:01.219050 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.219037    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:48:01.219050 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.219048    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-9487a-predictor-serving-cert\""
Jul 21 17:48:01.219379 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.219073    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2v5wd\""
Jul 21 17:48:01.219379 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.219014    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:48:01.224993 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.224971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"]
Jul 21 17:48:01.303096 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.303064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.303227 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.303105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-cabundle-cert\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.303227 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.303129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd55f48-6ecf-458e-9de5-71a8480d9f71-proxy-tls\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.303227 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.303199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kserve-provision-location\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.303342 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.303272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htph9\" (UniqueName: \"kubernetes.io/projected/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-api-access-htph9\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.403607 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.403576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-cabundle-cert\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.403778 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.403616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd55f48-6ecf-458e-9de5-71a8480d9f71-proxy-tls\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.403778 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.403642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kserve-provision-location\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.403778 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.403716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htph9\" (UniqueName: \"kubernetes.io/projected/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-api-access-htph9\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.403778 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.403739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.404109 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.404086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kserve-provision-location\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.404339 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.404323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-cabundle-cert\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.404386 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.404357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.406056 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.406035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd55f48-6ecf-458e-9de5-71a8480d9f71-proxy-tls\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.415135 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.415116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htph9\" (UniqueName: \"kubernetes.io/projected/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-api-access-htph9\") pod \"raw-sklearn-9487a-predictor-7ff69748bd-pfwzp\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") " pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.526716 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.526631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:01.685502 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:01.683192    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"]
Jul 21 17:48:01.688783 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:48:01.688759    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd55f48_6ecf_458e_9de5_71a8480d9f71.slice/crio-e8f9b8e16b7c03eb2cb21f1d0385aa29986f95fec4a5b2b4b25fd89eec94c744 WatchSource:0}: Error finding container e8f9b8e16b7c03eb2cb21f1d0385aa29986f95fec4a5b2b4b25fd89eec94c744: Status 404 returned error can't find the container with id e8f9b8e16b7c03eb2cb21f1d0385aa29986f95fec4a5b2b4b25fd89eec94c744
Jul 21 17:48:02.685284 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:02.685241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" event={"ID":"5cd55f48-6ecf-458e-9de5-71a8480d9f71","Type":"ContainerStarted","Data":"c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9"}
Jul 21 17:48:02.685691 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:02.685308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" event={"ID":"5cd55f48-6ecf-458e-9de5-71a8480d9f71","Type":"ContainerStarted","Data":"e8f9b8e16b7c03eb2cb21f1d0385aa29986f95fec4a5b2b4b25fd89eec94c744"}
Jul 21 17:48:05.703752 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:05.703645    2579 generic.go:356] "Generic (PLEG): container finished" podID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerID="c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9" exitCode=0
Jul 21 17:48:05.703752 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:05.703720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" event={"ID":"5cd55f48-6ecf-458e-9de5-71a8480d9f71","Type":"ContainerDied","Data":"c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9"}
Jul 21 17:48:06.710783 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:06.710742    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" event={"ID":"5cd55f48-6ecf-458e-9de5-71a8480d9f71","Type":"ContainerStarted","Data":"7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094"}
Jul 21 17:48:06.711164 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:06.710800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" event={"ID":"5cd55f48-6ecf-458e-9de5-71a8480d9f71","Type":"ContainerStarted","Data":"5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8"}
Jul 21 17:48:06.711164 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:06.711057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:06.711164 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:06.711097    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:06.712355 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:06.712332    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 17:48:06.733115 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:06.733075    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podStartSLOduration=2.099496891 podStartE2EDuration="5.733063872s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633566981s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:48:06.731296075 +0000 UTC m=+1312.290095061" watchObservedRunningTime="2026-07-21 17:48:06.733063872 +0000 UTC m=+1312.291862855"
Jul 21 17:48:07.715507 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:07.715473    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 17:48:12.720314 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:12.720286    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:48:12.720981 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:12.720951    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 17:48:22.721402 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:22.721365    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 17:48:32.721007 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:32.720958    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 17:48:42.721130 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:42.721094    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 17:48:52.721336 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:48:52.721292    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 17:49:02.721760 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:02.721719    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 17:49:12.721796 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:12.721756    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 17:49:19.003673 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:19.003631    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:49:21.308808 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:21.308774    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"]
Jul 21 17:49:21.309231 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:21.309002    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container" containerID="cri-o://5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8" gracePeriod=30
Jul 21 17:49:21.309231 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:21.309060    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kube-rbac-proxy" containerID="cri-o://7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094" gracePeriod=30
Jul 21 17:49:22.019898 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:22.019864    2579 generic.go:356] "Generic (PLEG): container finished" podID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerID="7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094" exitCode=2
Jul 21 17:49:22.020078 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:22.019937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" event={"ID":"5cd55f48-6ecf-458e-9de5-71a8480d9f71","Type":"ContainerDied","Data":"7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094"}
Jul 21 17:49:22.716027 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:22.715981    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 21 17:49:25.855216 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.855188    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:49:25.910917 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.910832    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd55f48-6ecf-458e-9de5-71a8480d9f71-proxy-tls\") pod \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") "
Jul 21 17:49:25.910917 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.910908    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kserve-provision-location\") pod \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") "
Jul 21 17:49:25.911138 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.910942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-rbac-proxy-sar-config\") pod \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") "
Jul 21 17:49:25.911138 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.910959    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-cabundle-cert\") pod \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") "
Jul 21 17:49:25.911138 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.911009    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htph9\" (UniqueName: \"kubernetes.io/projected/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-api-access-htph9\") pod \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\" (UID: \"5cd55f48-6ecf-458e-9de5-71a8480d9f71\") "
Jul 21 17:49:25.911297 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.911235    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cd55f48-6ecf-458e-9de5-71a8480d9f71" (UID: "5cd55f48-6ecf-458e-9de5-71a8480d9f71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:49:25.911345 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.911306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5cd55f48-6ecf-458e-9de5-71a8480d9f71" (UID: "5cd55f48-6ecf-458e-9de5-71a8480d9f71"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:49:25.911345 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.911324    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5cd55f48-6ecf-458e-9de5-71a8480d9f71" (UID: "5cd55f48-6ecf-458e-9de5-71a8480d9f71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:49:25.912956 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.912934    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-api-access-htph9" (OuterVolumeSpecName: "kube-api-access-htph9") pod "5cd55f48-6ecf-458e-9de5-71a8480d9f71" (UID: "5cd55f48-6ecf-458e-9de5-71a8480d9f71"). InnerVolumeSpecName "kube-api-access-htph9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:49:25.913070 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:25.913054    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd55f48-6ecf-458e-9de5-71a8480d9f71-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cd55f48-6ecf-458e-9de5-71a8480d9f71" (UID: "5cd55f48-6ecf-458e-9de5-71a8480d9f71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:49:26.012324 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.012293    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:49:26.012324 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.012317    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:49:26.012324 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.012328    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd55f48-6ecf-458e-9de5-71a8480d9f71-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:49:26.012522 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.012336    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htph9\" (UniqueName: \"kubernetes.io/projected/5cd55f48-6ecf-458e-9de5-71a8480d9f71-kube-api-access-htph9\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:49:26.012522 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.012345    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd55f48-6ecf-458e-9de5-71a8480d9f71-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:49:26.037259 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.037229    2579 generic.go:356] "Generic (PLEG): container finished" podID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerID="5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8" exitCode=0
Jul 21 17:49:26.037391 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.037286    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"
Jul 21 17:49:26.037391 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.037323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" event={"ID":"5cd55f48-6ecf-458e-9de5-71a8480d9f71","Type":"ContainerDied","Data":"5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8"}
Jul 21 17:49:26.037391 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.037375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp" event={"ID":"5cd55f48-6ecf-458e-9de5-71a8480d9f71","Type":"ContainerDied","Data":"e8f9b8e16b7c03eb2cb21f1d0385aa29986f95fec4a5b2b4b25fd89eec94c744"}
Jul 21 17:49:26.037523 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.037404    2579 scope.go:117] "RemoveContainer" containerID="7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094"
Jul 21 17:49:26.045535 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.045518    2579 scope.go:117] "RemoveContainer" containerID="5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8"
Jul 21 17:49:26.052633 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.052611    2579 scope.go:117] "RemoveContainer" containerID="c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9"
Jul 21 17:49:26.058750 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.058730    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"]
Jul 21 17:49:26.060510 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.060489    2579 scope.go:117] "RemoveContainer" containerID="7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094"
Jul 21 17:49:26.060854 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:49:26.060833    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094\": container with ID starting with 7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094 not found: ID does not exist" containerID="7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094"
Jul 21 17:49:26.060927 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.060861    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094"} err="failed to get container status \"7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094\": rpc error: code = NotFound desc = could not find container \"7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094\": container with ID starting with 7c7c0f0029fc951701b79b1ba26313318180b83e19b905c69f1ce240d9e4e094 not found: ID does not exist"
Jul 21 17:49:26.060927 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.060878    2579 scope.go:117] "RemoveContainer" containerID="5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8"
Jul 21 17:49:26.061164 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:49:26.061143    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8\": container with ID starting with 5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8 not found: ID does not exist" containerID="5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8"
Jul 21 17:49:26.061273 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.061170    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8"} err="failed to get container status \"5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8\": rpc error: code = NotFound desc = could not find container \"5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8\": container with ID starting with 5066b90f63eeda44aa48bfa17a3c6564a3507df68af0f3d31ae87dbc4b841ac8 not found: ID does not exist"
Jul 21 17:49:26.061273 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.061200    2579 scope.go:117] "RemoveContainer" containerID="c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9"
Jul 21 17:49:26.061442 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:49:26.061424    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9\": container with ID starting with c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9 not found: ID does not exist" containerID="c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9"
Jul 21 17:49:26.061492 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.061450    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9"} err="failed to get container status \"c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9\": rpc error: code = NotFound desc = could not find container \"c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9\": container with ID starting with c55545d92b261bd0fae26fcb1a9e643e70ad7a8e2a959ee89b00b2ad28f959c9 not found: ID does not exist"
Jul 21 17:49:26.062372 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.062353    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9487a-predictor-7ff69748bd-pfwzp"]
Jul 21 17:49:26.589867 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.589833    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"]
Jul 21 17:49:26.590175 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.590163    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kube-rbac-proxy"
Jul 21 17:49:26.590228 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.590177    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kube-rbac-proxy"
Jul 21 17:49:26.590228 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.590187    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="storage-initializer"
Jul 21 17:49:26.590228 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.590193    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="storage-initializer"
Jul 21 17:49:26.590228 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.590207    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container"
Jul 21 17:49:26.590228 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.590212    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container"
Jul 21 17:49:26.590400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.590269    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kube-rbac-proxy"
Jul 21 17:49:26.590400 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.590277    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" containerName="kserve-container"
Jul 21 17:49:26.594663 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.594630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.597606 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.597587    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-480dc-predictor-serving-cert\""
Jul 21 17:49:26.597733 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.597668    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:49:26.597799 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.597735    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:49:26.597799 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.597791    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-480dc-kube-rbac-proxy-sar-config\""
Jul 21 17:49:26.597983 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.597969    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2v5wd\""
Jul 21 17:49:26.600340 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.600319    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:49:26.600448 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.600378    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:49:26.604953 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.604930    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"]
Jul 21 17:49:26.718587 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.718556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67986832-04c1-48ba-b359-f4a1474e3c53-proxy-tls\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.718780 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.718601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-cabundle-cert\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.718780 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.718726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67986832-04c1-48ba-b359-f4a1474e3c53-kserve-provision-location\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.718780 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.718776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.718908 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.718794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8jl\" (UniqueName: \"kubernetes.io/projected/67986832-04c1-48ba-b359-f4a1474e3c53-kube-api-access-cw8jl\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.820028 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.819995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67986832-04c1-48ba-b359-f4a1474e3c53-proxy-tls\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.820183 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.820039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-cabundle-cert\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.820183 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.820079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67986832-04c1-48ba-b359-f4a1474e3c53-kserve-provision-location\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.820183 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.820112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.820183 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.820135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8jl\" (UniqueName: \"kubernetes.io/projected/67986832-04c1-48ba-b359-f4a1474e3c53-kube-api-access-cw8jl\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.820561 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.820539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67986832-04c1-48ba-b359-f4a1474e3c53-kserve-provision-location\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.820821 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.820802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.820955 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.820936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-cabundle-cert\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.822413 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.822397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67986832-04c1-48ba-b359-f4a1474e3c53-proxy-tls\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.829412 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.829387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8jl\" (UniqueName: \"kubernetes.io/projected/67986832-04c1-48ba-b359-f4a1474e3c53-kube-api-access-cw8jl\") pod \"raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:26.905333 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:26.905306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:27.004629 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:27.004595    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd55f48-6ecf-458e-9de5-71a8480d9f71" path="/var/lib/kubelet/pods/5cd55f48-6ecf-458e-9de5-71a8480d9f71/volumes"
Jul 21 17:49:27.039532 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:27.039507    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"]
Jul 21 17:49:27.041849 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:49:27.041822    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67986832_04c1_48ba_b359_f4a1474e3c53.slice/crio-d93530fab1c543132dc763df4c968fcb202b0e70ce60498c88245a5e14b6dc2a WatchSource:0}: Error finding container d93530fab1c543132dc763df4c968fcb202b0e70ce60498c88245a5e14b6dc2a: Status 404 returned error can't find the container with id d93530fab1c543132dc763df4c968fcb202b0e70ce60498c88245a5e14b6dc2a
Jul 21 17:49:28.049032 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:28.048997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" event={"ID":"67986832-04c1-48ba-b359-f4a1474e3c53","Type":"ContainerStarted","Data":"29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be"}
Jul 21 17:49:28.049419 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:28.049084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" event={"ID":"67986832-04c1-48ba-b359-f4a1474e3c53","Type":"ContainerStarted","Data":"d93530fab1c543132dc763df4c968fcb202b0e70ce60498c88245a5e14b6dc2a"}
Jul 21 17:49:31.063397 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:31.063363    2579 generic.go:356] "Generic (PLEG): container finished" podID="67986832-04c1-48ba-b359-f4a1474e3c53" containerID="29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be" exitCode=0
Jul 21 17:49:31.063890 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:31.063439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" event={"ID":"67986832-04c1-48ba-b359-f4a1474e3c53","Type":"ContainerDied","Data":"29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be"}
Jul 21 17:49:32.070520 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:32.070482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" event={"ID":"67986832-04c1-48ba-b359-f4a1474e3c53","Type":"ContainerStarted","Data":"417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b"}
Jul 21 17:49:32.070919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:32.070565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" event={"ID":"67986832-04c1-48ba-b359-f4a1474e3c53","Type":"ContainerStarted","Data":"c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c"}
Jul 21 17:49:32.070919 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:32.070765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:32.094368 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:32.094319    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podStartSLOduration=2.668559725 podStartE2EDuration="6.094306061s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.425746336s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:49:32.092746544 +0000 UTC m=+1397.651545532" watchObservedRunningTime="2026-07-21 17:49:32.094306061 +0000 UTC m=+1397.653105074"
Jul 21 17:49:33.076365 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:33.076338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:33.077687 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:33.077636    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 17:49:34.081235 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:34.081194    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 17:49:39.085693 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:39.085666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:49:39.086299 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:39.086270    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 17:49:49.086189 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:49.086148    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 17:49:59.086560 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:49:59.086523    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 17:50:09.086403 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:09.086316    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 17:50:19.086999 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:19.086962    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 17:50:29.086841 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:29.086794    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 17:50:39.086606 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:39.086566    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 17:50:49.087741 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:49.087712    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:50:56.884381 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:56.884347    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"]
Jul 21 17:50:56.884831 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:56.884568    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" containerID="cri-o://c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c" gracePeriod=30
Jul 21 17:50:56.884831 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:56.884613    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kube-rbac-proxy" containerID="cri-o://417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b" gracePeriod=30
Jul 21 17:50:57.421643 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:57.421611    2579 generic.go:356] "Generic (PLEG): container finished" podID="67986832-04c1-48ba-b359-f4a1474e3c53" containerID="417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b" exitCode=2
Jul 21 17:50:57.421836 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:57.421697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" event={"ID":"67986832-04c1-48ba-b359-f4a1474e3c53","Type":"ContainerDied","Data":"417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b"}
Jul 21 17:50:59.082180 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:59.082136    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 21 17:50:59.086468 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:50:59.086439    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 17:51:01.333799 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.333776    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:51:01.408590 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.408506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw8jl\" (UniqueName: \"kubernetes.io/projected/67986832-04c1-48ba-b359-f4a1474e3c53-kube-api-access-cw8jl\") pod \"67986832-04c1-48ba-b359-f4a1474e3c53\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") "
Jul 21 17:51:01.408590 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.408548    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-cabundle-cert\") pod \"67986832-04c1-48ba-b359-f4a1474e3c53\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") "
Jul 21 17:51:01.408590 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.408570    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-kube-rbac-proxy-sar-config\") pod \"67986832-04c1-48ba-b359-f4a1474e3c53\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") "
Jul 21 17:51:01.408923 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.408696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67986832-04c1-48ba-b359-f4a1474e3c53-kserve-provision-location\") pod \"67986832-04c1-48ba-b359-f4a1474e3c53\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") "
Jul 21 17:51:01.408923 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.408786    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67986832-04c1-48ba-b359-f4a1474e3c53-proxy-tls\") pod \"67986832-04c1-48ba-b359-f4a1474e3c53\" (UID: \"67986832-04c1-48ba-b359-f4a1474e3c53\") "
Jul 21 17:51:01.409031 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.408930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67986832-04c1-48ba-b359-f4a1474e3c53" (UID: "67986832-04c1-48ba-b359-f4a1474e3c53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:51:01.409031 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.408948    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67986832-04c1-48ba-b359-f4a1474e3c53" (UID: "67986832-04c1-48ba-b359-f4a1474e3c53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:51:01.409031 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.408974    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67986832-04c1-48ba-b359-f4a1474e3c53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67986832-04c1-48ba-b359-f4a1474e3c53" (UID: "67986832-04c1-48ba-b359-f4a1474e3c53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:51:01.409183 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.409087    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67986832-04c1-48ba-b359-f4a1474e3c53-kserve-provision-location\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:51:01.409183 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.409105    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-cabundle-cert\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:51:01.409183 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.409121    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67986832-04c1-48ba-b359-f4a1474e3c53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:51:01.410623 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.410593    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67986832-04c1-48ba-b359-f4a1474e3c53-kube-api-access-cw8jl" (OuterVolumeSpecName: "kube-api-access-cw8jl") pod "67986832-04c1-48ba-b359-f4a1474e3c53" (UID: "67986832-04c1-48ba-b359-f4a1474e3c53"). InnerVolumeSpecName "kube-api-access-cw8jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:51:01.410708 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.410669    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67986832-04c1-48ba-b359-f4a1474e3c53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67986832-04c1-48ba-b359-f4a1474e3c53" (UID: "67986832-04c1-48ba-b359-f4a1474e3c53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:51:01.440028 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.439998    2579 generic.go:356] "Generic (PLEG): container finished" podID="67986832-04c1-48ba-b359-f4a1474e3c53" containerID="c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c" exitCode=0
Jul 21 17:51:01.440156 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.440046    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"
Jul 21 17:51:01.440156 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.440074    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" event={"ID":"67986832-04c1-48ba-b359-f4a1474e3c53","Type":"ContainerDied","Data":"c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c"}
Jul 21 17:51:01.440156 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.440135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx" event={"ID":"67986832-04c1-48ba-b359-f4a1474e3c53","Type":"ContainerDied","Data":"d93530fab1c543132dc763df4c968fcb202b0e70ce60498c88245a5e14b6dc2a"}
Jul 21 17:51:01.440278 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.440162    2579 scope.go:117] "RemoveContainer" containerID="417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b"
Jul 21 17:51:01.449005 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.448989    2579 scope.go:117] "RemoveContainer" containerID="c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c"
Jul 21 17:51:01.457012 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.456995    2579 scope.go:117] "RemoveContainer" containerID="29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be"
Jul 21 17:51:01.463990 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.463972    2579 scope.go:117] "RemoveContainer" containerID="417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b"
Jul 21 17:51:01.464279 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:51:01.464260    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b\": container with ID starting with 417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b not found: ID does not exist" containerID="417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b"
Jul 21 17:51:01.464347 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.464286    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b"} err="failed to get container status \"417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b\": rpc error: code = NotFound desc = could not find container \"417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b\": container with ID starting with 417cf055437a321f60d464e7db06360de10fdf48fc6b5505aa2d2ae0a0059d5b not found: ID does not exist"
Jul 21 17:51:01.464347 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.464304    2579 scope.go:117] "RemoveContainer" containerID="c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c"
Jul 21 17:51:01.464606 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:51:01.464587    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c\": container with ID starting with c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c not found: ID does not exist" containerID="c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c"
Jul 21 17:51:01.464704 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.464613    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c"} err="failed to get container status \"c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c\": rpc error: code = NotFound desc = could not find container \"c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c\": container with ID starting with c385c63c34191829a374e23fc93b070b031eeea3a0f0fe1073f8fb3abdca193c not found: ID does not exist"
Jul 21 17:51:01.464704 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.464635    2579 scope.go:117] "RemoveContainer" containerID="29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be"
Jul 21 17:51:01.465276 ip-10-0-136-31 kubenswrapper[2579]: E0721 17:51:01.465251    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be\": container with ID starting with 29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be not found: ID does not exist" containerID="29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be"
Jul 21 17:51:01.465367 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.465280    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be"} err="failed to get container status \"29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be\": rpc error: code = NotFound desc = could not find container \"29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be\": container with ID starting with 29c939c1a2480af221ab279bb47ddf72189b49df69998bfe48251158c317f9be not found: ID does not exist"
Jul 21 17:51:01.467754 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.467698    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"]
Jul 21 17:51:01.469614 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.469595    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-480dc-predictor-59b8d4c96-f97kx"]
Jul 21 17:51:01.509593 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.509571    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67986832-04c1-48ba-b359-f4a1474e3c53-proxy-tls\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:51:01.509593 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:01.509592    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw8jl\" (UniqueName: \"kubernetes.io/projected/67986832-04c1-48ba-b359-f4a1474e3c53-kube-api-access-cw8jl\") on node \"ip-10-0-136-31.ec2.internal\" DevicePath \"\""
Jul 21 17:51:03.003546 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:03.003509    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" path="/var/lib/kubelet/pods/67986832-04c1-48ba-b359-f4a1474e3c53/volumes"
Jul 21 17:51:29.514041 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.514005    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tjhzd/must-gather-5h8nk"]
Jul 21 17:51:29.514449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.514323    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="storage-initializer"
Jul 21 17:51:29.514449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.514333    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="storage-initializer"
Jul 21 17:51:29.514449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.514343    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kube-rbac-proxy"
Jul 21 17:51:29.514449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.514349    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kube-rbac-proxy"
Jul 21 17:51:29.514449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.514363    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container"
Jul 21 17:51:29.514449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.514369    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container"
Jul 21 17:51:29.514449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.514422    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kserve-container"
Jul 21 17:51:29.514449 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.514429    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="67986832-04c1-48ba-b359-f4a1474e3c53" containerName="kube-rbac-proxy"
Jul 21 17:51:29.518235 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.518219    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjhzd/must-gather-5h8nk"
Jul 21 17:51:29.521328 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.521301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tjhzd\"/\"default-dockercfg-6djp4\""
Jul 21 17:51:29.522680 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.522642    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjhzd\"/\"openshift-service-ca.crt\""
Jul 21 17:51:29.522735 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.522670    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjhzd\"/\"kube-root-ca.crt\""
Jul 21 17:51:29.526227 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.526206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjhzd/must-gather-5h8nk"]
Jul 21 17:51:29.649621 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.649580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh9lh\" (UniqueName: \"kubernetes.io/projected/d39d7f18-044f-45ad-9420-0d9edd1c858b-kube-api-access-vh9lh\") pod \"must-gather-5h8nk\" (UID: \"d39d7f18-044f-45ad-9420-0d9edd1c858b\") " pod="openshift-must-gather-tjhzd/must-gather-5h8nk"
Jul 21 17:51:29.649828 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.649694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d39d7f18-044f-45ad-9420-0d9edd1c858b-must-gather-output\") pod \"must-gather-5h8nk\" (UID: \"d39d7f18-044f-45ad-9420-0d9edd1c858b\") " pod="openshift-must-gather-tjhzd/must-gather-5h8nk"
Jul 21 17:51:29.750084 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.750052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d39d7f18-044f-45ad-9420-0d9edd1c858b-must-gather-output\") pod \"must-gather-5h8nk\" (UID: \"d39d7f18-044f-45ad-9420-0d9edd1c858b\") " pod="openshift-must-gather-tjhzd/must-gather-5h8nk"
Jul 21 17:51:29.750230 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.750117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh9lh\" (UniqueName: \"kubernetes.io/projected/d39d7f18-044f-45ad-9420-0d9edd1c858b-kube-api-access-vh9lh\") pod \"must-gather-5h8nk\" (UID: \"d39d7f18-044f-45ad-9420-0d9edd1c858b\") " pod="openshift-must-gather-tjhzd/must-gather-5h8nk"
Jul 21 17:51:29.750442 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.750420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d39d7f18-044f-45ad-9420-0d9edd1c858b-must-gather-output\") pod \"must-gather-5h8nk\" (UID: \"d39d7f18-044f-45ad-9420-0d9edd1c858b\") " pod="openshift-must-gather-tjhzd/must-gather-5h8nk"
Jul 21 17:51:29.759291 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.759262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh9lh\" (UniqueName: \"kubernetes.io/projected/d39d7f18-044f-45ad-9420-0d9edd1c858b-kube-api-access-vh9lh\") pod \"must-gather-5h8nk\" (UID: \"d39d7f18-044f-45ad-9420-0d9edd1c858b\") " pod="openshift-must-gather-tjhzd/must-gather-5h8nk"
Jul 21 17:51:29.844794 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.844725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjhzd/must-gather-5h8nk"
Jul 21 17:51:29.971602 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.971576    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjhzd/must-gather-5h8nk"]
Jul 21 17:51:29.973925 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:51:29.973899    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd39d7f18_044f_45ad_9420_0d9edd1c858b.slice/crio-660b87ed9cde2cda5c67c25315e16bf29a0831be7c9a69f85c03a215e1a7364e WatchSource:0}: Error finding container 660b87ed9cde2cda5c67c25315e16bf29a0831be7c9a69f85c03a215e1a7364e: Status 404 returned error can't find the container with id 660b87ed9cde2cda5c67c25315e16bf29a0831be7c9a69f85c03a215e1a7364e
Jul 21 17:51:29.975782 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:29.975767    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:51:30.558897 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:30.558860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjhzd/must-gather-5h8nk" event={"ID":"d39d7f18-044f-45ad-9420-0d9edd1c858b","Type":"ContainerStarted","Data":"660b87ed9cde2cda5c67c25315e16bf29a0831be7c9a69f85c03a215e1a7364e"}
Jul 21 17:51:31.566997 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:31.566956    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjhzd/must-gather-5h8nk" event={"ID":"d39d7f18-044f-45ad-9420-0d9edd1c858b","Type":"ContainerStarted","Data":"abb44f319018de429e16e3ad07508b754dca903c2868811ef9b990b9d17cd2c8"}
Jul 21 17:51:31.571016 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:31.567012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjhzd/must-gather-5h8nk" event={"ID":"d39d7f18-044f-45ad-9420-0d9edd1c858b","Type":"ContainerStarted","Data":"c1a66f064499a48acbdb129d6012330ec429a393fd00182bf4625c6fc67de646"}
Jul 21 17:51:31.586624 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:31.586555    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tjhzd/must-gather-5h8nk" podStartSLOduration=1.770678733 podStartE2EDuration="2.586534159s" totalImagesPullingTime="815.855426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:51:31.584535279 +0000 UTC m=+1517.143334264" watchObservedRunningTime="2026-07-21 17:51:31.586534159 +0000 UTC m=+1517.145333144"
Jul 21 17:51:32.220034 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:32.220001    2579 ???:1] "http: TLS handshake error from 10.0.139.6:38588: EOF"
Jul 21 17:51:32.225096 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:32.225058    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l5mfk_1511f43e-7fd8-4f20-86ca-f9eb3c032ac5/global-pull-secret-syncer/0.log"
Jul 21 17:51:32.340968 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:32.340935    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p87zq_b0fb0ecb-7a4d-4a2f-b7f1-2b75b2a4edb1/konnectivity-agent/0.log"
Jul 21 17:51:32.423703 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:32.423676    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-31.ec2.internal_c1faed2aff0ab49640344e2b4b87931f/haproxy/0.log"
Jul 21 17:51:36.514923 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:36.514882    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_185f77f4-1642-4f61-8dfc-b6d40ec472f3/alertmanager/0.log"
Jul 21 17:51:36.544424 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:36.544376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_185f77f4-1642-4f61-8dfc-b6d40ec472f3/config-reloader/0.log"
Jul 21 17:51:36.572768 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:36.572727    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_185f77f4-1642-4f61-8dfc-b6d40ec472f3/kube-rbac-proxy-web/0.log"
Jul 21 17:51:36.596093 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:36.596060    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_185f77f4-1642-4f61-8dfc-b6d40ec472f3/kube-rbac-proxy/0.log"
Jul 21 17:51:36.622499 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:36.622471    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_185f77f4-1642-4f61-8dfc-b6d40ec472f3/kube-rbac-proxy-metric/0.log"
Jul 21 17:51:36.646322 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:36.646295    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_185f77f4-1642-4f61-8dfc-b6d40ec472f3/prom-label-proxy/0.log"
Jul 21 17:51:36.669488 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:36.669344    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_185f77f4-1642-4f61-8dfc-b6d40ec472f3/init-config-reloader/0.log"
Jul 21 17:51:36.849145 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:36.849111    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-crfrn_76603123-4844-43ff-bedf-472ce2c72a9c/monitoring-plugin/0.log"
Jul 21 17:51:36.965767 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:36.965684    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhjhd_b71adf0f-c005-45ea-a6a9-b4684095d877/node-exporter/0.log"
Jul 21 17:51:36.988866 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:36.988840    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhjhd_b71adf0f-c005-45ea-a6a9-b4684095d877/kube-rbac-proxy/0.log"
Jul 21 17:51:37.014237 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:37.014204    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhjhd_b71adf0f-c005-45ea-a6a9-b4684095d877/init-textfile/0.log"
Jul 21 17:51:37.410464 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:37.410419    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-d759z_cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9/prometheus-operator/0.log"
Jul 21 17:51:37.433614 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:37.433570    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-d759z_cc0ae0ca-320d-48bc-9ba8-b5f68ef6b5a9/kube-rbac-proxy/0.log"
Jul 21 17:51:39.992405 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:39.992370    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b454c7bc8-gqqgd_2b37ae05-94f4-4157-a8b5-c612186acd24/console/0.log"
Jul 21 17:51:40.024148 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.024122    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-c4rnj_6c484675-4f71-4113-a522-e411348e951b/download-server/0.log"
Jul 21 17:51:40.388727 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.388685    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"]
Jul 21 17:51:40.392549 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.392523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.405312 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.405277    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"]
Jul 21 17:51:40.450227 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.450185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-podres\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.450402 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.450237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-proc\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.450402 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.450299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phk85\" (UniqueName: \"kubernetes.io/projected/3f616f13-b607-4652-a716-02152fba3b54-kube-api-access-phk85\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.450505 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.450418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-lib-modules\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.450505 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.450447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-sys\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.551017 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.550983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-proc\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.551202 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.551034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phk85\" (UniqueName: \"kubernetes.io/projected/3f616f13-b607-4652-a716-02152fba3b54-kube-api-access-phk85\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.551202 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.551112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-proc\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.551202 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.551117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-lib-modules\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.551202 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.551187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-sys\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.551427 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.551232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-podres\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.551427 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.551243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-lib-modules\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.551427 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.551318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-sys\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.551427 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.551340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f616f13-b607-4652-a716-02152fba3b54-podres\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.559609 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.559582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phk85\" (UniqueName: \"kubernetes.io/projected/3f616f13-b607-4652-a716-02152fba3b54-kube-api-access-phk85\") pod \"perf-node-gather-daemonset-rjgz7\" (UID: \"3f616f13-b607-4652-a716-02152fba3b54\") " pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.561503 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.561484    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-jbwcp_42597b38-e5e4-42f6-8599-3d2d5645dbf5/volume-data-source-validator/0.log"
Jul 21 17:51:40.705684 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.705577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:40.865261 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:40.865237    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"]
Jul 21 17:51:40.868684 ip-10-0-136-31 kubenswrapper[2579]: W0721 17:51:40.868329    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3f616f13_b607_4652_a716_02152fba3b54.slice/crio-4b212f519b7c1548d12241e447b7c36558f17481417ca09d542489a7579b6e89 WatchSource:0}: Error finding container 4b212f519b7c1548d12241e447b7c36558f17481417ca09d542489a7579b6e89: Status 404 returned error can't find the container with id 4b212f519b7c1548d12241e447b7c36558f17481417ca09d542489a7579b6e89
Jul 21 17:51:41.478606 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:41.478569    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ccjk8_20be539f-d29f-4ed6-8c74-1635fdea9bd3/dns/0.log"
Jul 21 17:51:41.501078 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:41.501043    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ccjk8_20be539f-d29f-4ed6-8c74-1635fdea9bd3/kube-rbac-proxy/0.log"
Jul 21 17:51:41.635714 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:41.635679    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7" event={"ID":"3f616f13-b607-4652-a716-02152fba3b54","Type":"ContainerStarted","Data":"f33a3a26a089debb8ab6f6804b2dec75b34279d2d332f3bdd86f92fd6d149be7"}
Jul 21 17:51:41.635884 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:41.635739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7" event={"ID":"3f616f13-b607-4652-a716-02152fba3b54","Type":"ContainerStarted","Data":"4b212f519b7c1548d12241e447b7c36558f17481417ca09d542489a7579b6e89"}
Jul 21 17:51:41.635884 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:41.635810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:41.653245 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:41.653194    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7" podStartSLOduration=1.653179084 podStartE2EDuration="1.653179084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:51:41.651747399 +0000 UTC m=+1527.210546384" watchObservedRunningTime="2026-07-21 17:51:41.653179084 +0000 UTC m=+1527.211978046"
Jul 21 17:51:41.679558 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:41.679531    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mz828_3b03ab87-1a4b-4b9d-b530-e3c8d082464d/dns-node-resolver/0.log"
Jul 21 17:51:42.194120 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:42.194093    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sckv6_59fc317b-41ba-4e6e-8b7c-5ae9112a89cf/node-ca/0.log"
Jul 21 17:51:42.897756 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:42.897701    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9f678b54d-lmmtf_e56663cb-5de7-4266-b7ca-1aa0bf94928a/router/0.log"
Jul 21 17:51:43.287097 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:43.287021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gclnx_7000eef1-3ae1-4925-950d-4929f1c689d1/serve-healthcheck-canary/0.log"
Jul 21 17:51:43.634307 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:43.634277    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-x2sft_8677903c-34d1-41e8-b0c7-be1e3188b4ce/insights-operator/0.log"
Jul 21 17:51:43.634470 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:43.634461    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-x2sft_8677903c-34d1-41e8-b0c7-be1e3188b4ce/insights-operator/1.log"
Jul 21 17:51:43.726190 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:43.726161    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v92lz_10f013b0-411e-4cd7-b170-2dd308190cea/kube-rbac-proxy/0.log"
Jul 21 17:51:43.749345 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:43.749315    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v92lz_10f013b0-411e-4cd7-b170-2dd308190cea/exporter/0.log"
Jul 21 17:51:43.772008 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:43.771982    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v92lz_10f013b0-411e-4cd7-b170-2dd308190cea/extractor/0.log"
Jul 21 17:51:46.088574 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:46.088543    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6b69bf598b-f697k_41c976e9-cec7-4777-a011-0b2987f4fc2a/manager/0.log"
Jul 21 17:51:46.139534 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:46.139507    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-5l5zg_1389af9a-5961-4272-b30b-83a03a4e70aa/server/0.log"
Jul 21 17:51:46.299170 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:46.299141    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-48vr9_d29aa7fb-c251-4e97-bad0-0a72d147d211/seaweedfs/0.log"
Jul 21 17:51:47.650040 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:47.650010    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tjhzd/perf-node-gather-daemonset-rjgz7"
Jul 21 17:51:50.707757 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:50.707722    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-prmlj_3f5d4bf0-deac-486f-a3dd-a5ec092c342d/kube-storage-version-migrator-operator/1.log"
Jul 21 17:51:50.709074 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:50.709045    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-prmlj_3f5d4bf0-deac-486f-a3dd-a5ec092c342d/kube-storage-version-migrator-operator/0.log"
Jul 21 17:51:51.712214 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:51.712188    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-67qsr_5dad033f-a636-412a-81fc-3e7b683e14e8/kube-multus-additional-cni-plugins/0.log"
Jul 21 17:51:51.736681 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:51.736639    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-67qsr_5dad033f-a636-412a-81fc-3e7b683e14e8/egress-router-binary-copy/0.log"
Jul 21 17:51:51.762265 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:51.762186    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-67qsr_5dad033f-a636-412a-81fc-3e7b683e14e8/cni-plugins/0.log"
Jul 21 17:51:51.788605 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:51.788550    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-67qsr_5dad033f-a636-412a-81fc-3e7b683e14e8/bond-cni-plugin/0.log"
Jul 21 17:51:51.811840 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:51.811810    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-67qsr_5dad033f-a636-412a-81fc-3e7b683e14e8/routeoverride-cni/0.log"
Jul 21 17:51:51.835117 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:51.835089    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-67qsr_5dad033f-a636-412a-81fc-3e7b683e14e8/whereabouts-cni-bincopy/0.log"
Jul 21 17:51:51.860752 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:51.860727    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-67qsr_5dad033f-a636-412a-81fc-3e7b683e14e8/whereabouts-cni/0.log"
Jul 21 17:51:52.239333 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:52.239304    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qgmkm_5856df40-592b-4d35-becf-d70792f8e5cf/kube-multus/0.log"
Jul 21 17:51:52.259828 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:52.259791    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f6xmv_a3f8ea96-7aec-48b2-85ef-6fe96c80ce62/network-metrics-daemon/0.log"
Jul 21 17:51:52.281591 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:52.281566    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f6xmv_a3f8ea96-7aec-48b2-85ef-6fe96c80ce62/kube-rbac-proxy/0.log"
Jul 21 17:51:53.331379 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:53.331353    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mr248_af64258e-d822-4ebb-b7a4-9ec5a1b71267/ovn-controller/0.log"
Jul 21 17:51:53.349737 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:53.349711    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mr248_af64258e-d822-4ebb-b7a4-9ec5a1b71267/ovn-acl-logging/0.log"
Jul 21 17:51:53.373046 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:53.373016    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mr248_af64258e-d822-4ebb-b7a4-9ec5a1b71267/kube-rbac-proxy-node/0.log"
Jul 21 17:51:53.396681 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:53.396643    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mr248_af64258e-d822-4ebb-b7a4-9ec5a1b71267/kube-rbac-proxy-ovn-metrics/0.log"
Jul 21 17:51:53.421490 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:53.421462    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mr248_af64258e-d822-4ebb-b7a4-9ec5a1b71267/northd/0.log"
Jul 21 17:51:53.443595 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:53.443572    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mr248_af64258e-d822-4ebb-b7a4-9ec5a1b71267/nbdb/0.log"
Jul 21 17:51:53.464886 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:53.464864    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mr248_af64258e-d822-4ebb-b7a4-9ec5a1b71267/sbdb/0.log"
Jul 21 17:51:53.509792 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:53.509766    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mr248_af64258e-d822-4ebb-b7a4-9ec5a1b71267/ovnkube-controller/0.log"
Jul 21 17:51:54.759597 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:54.759564    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8vgqn_53d47a1a-f442-4cac-abd3-941e47645e03/network-check-target-container/0.log"
Jul 21 17:51:55.631759 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:55.631731    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-c5m62_add844a8-7efd-4d4f-9dd3-535fbae49877/iptables-alerter/0.log"
Jul 21 17:51:56.320722 ip-10-0-136-31 kubenswrapper[2579]: I0721 17:51:56.320698    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-7wqqh_0a4707f9-96d8-41ac-9c10-09fa08d73e89/tuned/0.log"