Jul 10 18:56:06.380904 ip-10-0-136-15 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 10 18:56:06.380920 ip-10-0-136-15 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 10 18:56:06.380930 ip-10-0-136-15 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 10 18:56:06.381265 ip-10-0-136-15 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 10 18:56:16.629811 ip-10-0-136-15 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 10 18:56:16.629830 ip-10-0-136-15 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0ceb4403f4df4b77b3bf4ed138ce4641 --
Jul 10 18:58:14.784863 ip-10-0-136-15 systemd[1]: Starting Kubernetes Kubelet...
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.194506    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.198422    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.198434    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.198438    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.198441    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.198444    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.198447    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.198450    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.198453    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:15.225116 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.198456    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199933    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199939    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199942    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199946    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199949    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199951    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199954    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199957    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199960    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199962    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199965    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199968    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199970    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199974    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199976    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199979    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199981    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199984    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199987    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:15.233054 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199990    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.199997    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200000    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200003    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200005    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200008    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200011    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200013    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200016    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200018    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200021    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200024    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200026    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200029    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200032    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200035    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200037    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200040    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200042    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:15.233930 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200046    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200051    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200054    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200057    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200061    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200065    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200068    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200071    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200074    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200078    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200081    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200084    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200087    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200090    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200093    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200095    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200098    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200101    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200103    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200106    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:15.234814 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200108    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200111    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200114    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200116    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200119    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200122    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200124    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200128    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200131    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200134    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200137    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200140    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200143    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200145    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200148    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200152    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200154    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200157    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200159    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:15.235725 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200162    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200165    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200167    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200170    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200173    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200175    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200178    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200180    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200184    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200187    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200190    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200192    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.200195    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201759    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201768    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201771    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201775    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201778    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201782    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201785    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:15.236612 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201788    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201791    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201794    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201797    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201800    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201803    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201806    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201808    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201811    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201814    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201816    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201821    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201824    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201828    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201831    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201833    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201836    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201839    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201841    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201844    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:15.237904 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201846    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201849    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201852    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201854    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201857    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201859    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201862    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201864    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201867    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201869    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201872    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201875    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201878    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201880    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201883    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201885    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201888    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201892    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201895    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201897    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:15.238774 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201900    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201903    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201906    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201909    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201911    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201914    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201916    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201919    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201930    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201934    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201937    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201941    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201944    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201948    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201952    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201956    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201960    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201962    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201965    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201968    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:15.239844 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201973    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201976    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201980    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201984    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201990    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.201997    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202002    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202007    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202011    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202017    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202021    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202024    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202028    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202034    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202037    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202041    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202045    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202049    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202053    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:15.240733 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202058    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202062    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202066    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202070    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202073    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202077    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202081    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202085    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202089    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202092    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202097    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202100    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202106    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202613    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202620    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202624    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202628    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202639    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202643    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202647    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:15.241624 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202651    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202654    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202660    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202664    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202668    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202672    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202678    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202682    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202685    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202689    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202693    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202696    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202700    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202704    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202707    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202711    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202715    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202719    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202722    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202726    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:15.242500 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202730    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202733    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202737    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202741    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202745    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202749    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202753    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202756    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202760    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202763    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202774    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202779    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202784    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202787    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202793    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202796    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202801    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202804    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202809    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202813    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:15.243582 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202816    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202820    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202824    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202827    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202831    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202835    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202839    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202843    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202846    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202850    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202854    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202858    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202861    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202865    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202868    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202873    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202876    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202880    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202884    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:15.244472 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202887    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202891    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202894    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202898    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202905    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202909    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202913    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202918    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202922    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202926    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202929    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202938    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202942    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202946    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202949    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202953    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202957    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202961    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202964    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202968    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:15.245331 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202972    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202976    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202983    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202988    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202992    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.202996    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.203000    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.203004    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.203010    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.203015    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.203019    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.203024    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.203028    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203111    2567 flags.go:64] FLAG: --address="0.0.0.0"
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203120    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203129    2567 flags.go:64] FLAG: --anonymous-auth="true"
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203136    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203142    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203146    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203153    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 10 18:58:15.246220 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203159    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203164    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203168    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203175    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203180    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203185    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203188    2567 flags.go:64] FLAG: --cgroup-root=""
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203193    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203197    2567 flags.go:64] FLAG: --client-ca-file=""
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203200    2567 flags.go:64] FLAG: --cloud-provider="external"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203205    2567 flags.go:64] FLAG: --cluster-dns="[]"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203210    2567 flags.go:64] FLAG: --cluster-domain=""
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203214    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203218    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203223    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203228    2567 flags.go:64] FLAG: --container-log-max-files="5"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203232    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203239    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203248    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203253    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203257    2567 flags.go:64] FLAG: --contention-profiling="false"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203261    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203265    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203270    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 10 18:58:15.247090 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203275    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203279    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203283    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203287    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203291    2567 flags.go:64] FLAG: --enable-load-reader="false"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203295    2567 flags.go:64] FLAG: --enable-server="true"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203300    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203308    2567 flags.go:64] FLAG: --event-burst="100"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203312    2567 flags.go:64] FLAG: --event-qps="50"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203315    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203320    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203326    2567 flags.go:64] FLAG: --eviction-hard=""
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203331    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203335    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203339    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203343    2567 flags.go:64] FLAG: --eviction-soft=""
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203347    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203352    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203356    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203360    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203364    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203368    2567 flags.go:64] FLAG: --fail-swap-on="true"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203372    2567 flags.go:64] FLAG: --feature-gates=""
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203377    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203381    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 10 18:58:15.248195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203385    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203389    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203393    2567 flags.go:64] FLAG: --healthz-port="10248"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203397    2567 flags.go:64] FLAG: --help="false"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203401    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-136-15.ec2.internal"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203406    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203409    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203413    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203418    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203423    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203427    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203431    2567 flags.go:64] FLAG: --image-service-endpoint=""
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203435    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203439    2567 flags.go:64] FLAG: --kube-api-burst="100"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203445    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203450    2567 flags.go:64] FLAG: --kube-api-qps="50"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203454    2567 flags.go:64] FLAG: --kube-reserved=""
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203458    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203464    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203469    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203473    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203477    2567 flags.go:64] FLAG: --lock-file=""
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203481    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203485    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 10 18:58:15.249488 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203490    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203496    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203500    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203505    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203509    2567 flags.go:64] FLAG: --logging-format="text"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203513    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203518    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203522    2567 flags.go:64] FLAG: --manifest-url=""
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203527    2567 flags.go:64] FLAG: --manifest-url-header=""
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203532    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203537    2567 flags.go:64] FLAG: --max-open-files="1000000"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203549    2567 flags.go:64] FLAG: --max-pods="110"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203553    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203557    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203561    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203565    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203569    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203574    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203578    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203601    2567 flags.go:64] FLAG: --node-status-max-images="50"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203606    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203610    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203616    2567 flags.go:64] FLAG: --pod-cidr=""
Jul 10 18:58:15.250613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203620    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203627    2567 flags.go:64] FLAG: --pod-manifest-path=""
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203631    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203637    2567 flags.go:64] FLAG: --pods-per-core="0"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203641    2567 flags.go:64] FLAG: --port="10250"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203645    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203649    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b0e3878f9d26e1d8"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203653    2567 flags.go:64] FLAG: --qos-reserved=""
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203659    2567 flags.go:64] FLAG: --read-only-port="10255"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203663    2567 flags.go:64] FLAG: --register-node="true"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203667    2567 flags.go:64] FLAG: --register-with-taints=""
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203672    2567 flags.go:64] FLAG: --registry-burst="10"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203676    2567 flags.go:64] FLAG: --registry-qps="5"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203680    2567 flags.go:64] FLAG: --reserved-cpus=""
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203684    2567 flags.go:64] FLAG: --reserved-memory=""
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203689    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203694    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203698    2567 flags.go:64] FLAG: --rotate-certificates="false"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203703    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203706    2567 flags.go:64] FLAG: --runonce="false"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203709    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203714    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203718    2567 flags.go:64] FLAG: --seccomp-default="false"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203722    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203726    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203730    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 10 18:58:15.251915 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203734    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203739    2567 flags.go:64] FLAG: --storage-driver-password="root"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203743    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203747    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203751    2567 flags.go:64] FLAG: --storage-driver-user="root"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203755    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203761    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203765    2567 flags.go:64] FLAG: --system-cgroups=""
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203769    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203778    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203783    2567 flags.go:64] FLAG: --tls-cert-file=""
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203788    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203793    2567 flags.go:64] FLAG: --tls-min-version=""
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203797    2567 flags.go:64] FLAG: --tls-private-key-file=""
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203802    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203806    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203811    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203816    2567 flags.go:64] FLAG: --v="2"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203821    2567 flags.go:64] FLAG: --version="false"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203827    2567 flags.go:64] FLAG: --vmodule=""
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203831    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.203836    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.203989    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.203995    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:15.323427 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.203999    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:15.306001 ip-10-0-136-15 systemd[1]: Started Kubernetes Kubelet.
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204001    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204004    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204008    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204018    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204022    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204025    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204028    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204031    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204034    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204037    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204040    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204042    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204045    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204050    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204052    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204056    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204060    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204063    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204066    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:15.324982 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204069    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204071    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204073    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204076    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204078    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204081    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204083    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204086    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204088    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204091    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204093    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204096    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204098    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204101    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204104    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204106    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204109    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204111    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204114    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204116    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:15.333215 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204119    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204121    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204124    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204127    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204130    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204133    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204138    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204141    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204143    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204148    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204151    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204153    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204156    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204158    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204161    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204163    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204166    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204169    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204171    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:15.334333 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204174    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204176    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204179    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204181    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204184    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204187    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204189    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204192    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204195    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204197    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204200    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204202    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204205    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204207    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204210    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204212    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204215    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204217    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204220    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204225    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:15.335222 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204227    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204230    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204234    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204237    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204240    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204244    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204247    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204250    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204253    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204256    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204258    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204261    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204263    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204266    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204268    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204271    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204273    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:15.336326 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.204276    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.205053    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.211203    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.211218    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.211234    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.211244    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211298    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211302    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211307    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211311    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211314    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211317    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211320    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211323    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:15.337210 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211325    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211328    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211331    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211334    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211336    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211339    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211342    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211344    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211347    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211350    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211353    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211357    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211360    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211364    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211367    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211370    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211374    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211377    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211380    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211382    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:15.338079 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211385    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211388    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211391    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211394    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211396    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211399    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211401    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211404    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211406    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211409    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211411    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211414    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211417    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211419    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211422    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211424    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211427    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211429    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211432    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211434    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:15.338994 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211437    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211439    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211442    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211444    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211447    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211449    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211452    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211454    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211458    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211461    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211464    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211466    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211468    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211471    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211474    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211476    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211479    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211481    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211484    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:15.339924 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211487    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211489    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211492    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211494    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211497    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211499    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211502    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211505    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211507    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211510    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211513    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211515    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211519    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211521    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211525    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211528    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211531    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211533    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211536    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211539    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:15.340822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211542    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211546    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211548    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211551    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211553    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211556    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211558    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211561    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211563    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211566    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211569    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211571    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.211576    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211703    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211708    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:15.341741 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211711    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211713    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211716    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211719    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211723    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211728    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211732    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211735    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211742    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211745    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211748    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211751    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211753    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211756    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211759    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211761    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211764    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211766    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211769    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211774    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:15.342432 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211777    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211779    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211782    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211784    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211787    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211790    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211792    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211795    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211798    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211800    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211803    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211805    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211808    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211810    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211812    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211815    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211817    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211820    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211823    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:15.343539 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211825    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211829    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211831    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211834    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211836    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211839    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211843    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211846    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211849    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211852    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211855    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211858    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211862    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211864    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211867    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211870    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211872    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211875    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211877    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:15.344425 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211880    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211883    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211885    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211888    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211891    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211893    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211896    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211898    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211901    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211903    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211906    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211908    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211911    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211914    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211917    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211935    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211939    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211943    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211946    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211949    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:15.345323 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211951    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211955    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211957    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211960    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211962    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211966    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211969    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211971    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211974    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211976    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211979    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211981    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211984    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211986    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211989    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211992    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211995    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211997    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:15.346211 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:15.211999    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.212004    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.212118    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.214175    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.215358    2567 server.go:1019] "Starting client certificate rotation"
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.215453    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.216104    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.217623    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.248447    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.259303    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.272008    2567 log.go:25] "Validated CRI v1 runtime API"
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.273939    2567 log.go:25] "Validated CRI v1 image API"
Jul 10 18:58:15.347082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.274782    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 10 18:58:15.347954 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.279063    2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9b0a78f3-2632-4dfc-a0f3-200442fc2ea1:/dev/nvme0n1p3 bb109d24-685e-40c1-94d6-a48e1bcec680:/dev/nvme0n1p4]
Jul 10 18:58:15.347954 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.279079    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 10 18:58:15.347954 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.279114    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tchks"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.284041    2567 manager.go:216] Machine: {Timestamp:2026-07-10 18:58:15.282732838 +0000 UTC m=+0.379700495 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099018 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eb3152934e32a8f7c9ff4d31c7102 SystemUUID:ec2eb315-2934-e32a-8f7c-9ff4d31c7102 BootID:0ceb4403-f4df-4b77-b3bf-4ed138ce4641 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:81:ac:54:df:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:81:ac:54:df:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:3a:bd:01:d9:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.284145    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.284218    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.284559    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tchks"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.287642    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.287673    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.288418    2567 topology_manager.go:138] "Creating topology manager with none policy"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.288427    2567 container_manager_linux.go:309] "Creating device plugin manager"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.288434    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.289871    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.290904    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.293964    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.294076    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.294098    2567 kubelet.go:486] "Attempting to sync node with API server"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.294116    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.294129    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 10 18:58:15.348612 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.294138    2567 kubelet.go:398] "Adding apiserver pod source"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.294148    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.294500    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.294519    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.297578    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.299407    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.299438    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.300998    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301019    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301029    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301037    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301050    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301067    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301097    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301106    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301117    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301125    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301138    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301146    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301915    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.301936    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.305162    2567 server.go:1268] "Started kubelet"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.305235    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.305296    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.306527    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.308201    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.308372    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.308508    2567 server.go:310] "Adding debug handlers to kubelet server"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.311559    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:15.311974    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.312975    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.312988    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:15.313050    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-15.ec2.internal\" not found"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.313486    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.313502    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.313489    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.313582    2567 reconstruct.go:97] "Volume reconstruction finished"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.313603    2567 reconciler.go:29] "Reconciler: start to sync state"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.313775    2567 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-15.ec2.internal" not found
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.314114    2567 factory.go:153] Registering CRI-O factory
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.314128    2567 factory.go:223] Registration of the crio container factory successfully
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.314180    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.314193    2567 factory.go:55] Registering systemd factory
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.314200    2567 factory.go:223] Registration of the systemd container factory successfully
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.314221    2567 factory.go:103] Registering Raw factory
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.314231    2567 manager.go:1196] Started watching for new ooms in manager
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.314661    2567 manager.go:349] Starting recovery of all containers
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.314795    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:15.317207    2567 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-15.ec2.internal\" not found" node="ip-10-0-136-15.ec2.internal"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.325604    2567 manager.go:354] Recovery completed
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.328728    2567 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-15.ec2.internal" not found
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.329649    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:15.349478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.331409    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-15.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.331433    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-15.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.331442    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-15.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.331934    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.331943    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.331962    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.331977    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.332730    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-15.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.332755    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-15.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.332766    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-15.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.333125    2567 policy_none.go:49] "None policy: Start"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.333137    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.333146    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.334279    2567 policy_none.go:47] "Start"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:15.387646    2567 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-15.ec2.internal" not found
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:15.413545    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-15.ec2.internal\" not found"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.279540    2567 manager.go:337] "Starting Device Plugin manager"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:17.279564    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.279572    2567 server.go:87] "Starting device plugin registration server"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.279792    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.279803    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.279892    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.280062    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.280187    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:17.280399    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 10 18:58:17.284247 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:17.280440    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-15.ec2.internal\" not found"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.380236    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.381070    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-15.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.381096    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-15.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.381107    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-15.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.381129    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.390350    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.411528    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.412937    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.412975    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.412994    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:17.413021    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 10 18:58:17.493248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.415260    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:17.645073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.513340    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal"]
Jul 10 18:58:17.645073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.514606    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.645073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.515401    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.645073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.548462    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.645073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.552838    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.645073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.560247    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:58:17.645073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.564010    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:58:17.645073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.566180    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b86e7df3f891ef5281cf18ca965a9f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal\" (UID: \"3b86e7df3f891ef5281cf18ca965a9f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.645073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.566212    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d18d246e0bedf5b391d12d0230022e47-config\") pod \"kube-apiserver-proxy-ip-10-0-136-15.ec2.internal\" (UID: \"d18d246e0bedf5b391d12d0230022e47\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.645073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.566229    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b86e7df3f891ef5281cf18ca965a9f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal\" (UID: \"3b86e7df3f891ef5281cf18ca965a9f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.757669 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.666730    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b86e7df3f891ef5281cf18ca965a9f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal\" (UID: \"3b86e7df3f891ef5281cf18ca965a9f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.757669 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.666751    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b86e7df3f891ef5281cf18ca965a9f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal\" (UID: \"3b86e7df3f891ef5281cf18ca965a9f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.757669 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.666769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d18d246e0bedf5b391d12d0230022e47-config\") pod \"kube-apiserver-proxy-ip-10-0-136-15.ec2.internal\" (UID: \"d18d246e0bedf5b391d12d0230022e47\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.757669 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.666807    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d18d246e0bedf5b391d12d0230022e47-config\") pod \"kube-apiserver-proxy-ip-10-0-136-15.ec2.internal\" (UID: \"d18d246e0bedf5b391d12d0230022e47\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.757669 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.666817    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b86e7df3f891ef5281cf18ca965a9f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal\" (UID: \"3b86e7df3f891ef5281cf18ca965a9f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.757669 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.666851    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b86e7df3f891ef5281cf18ca965a9f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal\" (UID: \"3b86e7df3f891ef5281cf18ca965a9f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.868965 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.862448    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:17.993839 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:17.884577    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal"
Jul 10 18:58:18.116822 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.061182    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 10 18:58:18.116822 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.061296    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:58:18.116822 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.061306    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:58:18.116822 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.061306    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.135867    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:53:17 +0000 UTC" deadline="2028-02-15 19:39:54.532618194 +0000 UTC"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.135901    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14040h41m36.396719228s"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.139753    2567 apiserver.go:52] "Watching apiserver"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.153519    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.153835    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-6wg9f","openshift-dns/node-resolver-4xtcp","openshift-multus/multus-5hnlc","openshift-multus/multus-additional-cni-plugins-8m2j6","openshift-multus/network-metrics-daemon-cbt5f","openshift-network-operator/iptables-alerter-72vr9","kube-system/global-pull-secret-syncer-4xprr","kube-system/konnectivity-agent-64zsj","openshift-image-registry/node-ca-m9r7c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal","openshift-network-diagnostics/network-check-target-mgkkh","openshift-ovn-kubernetes/ovnkube-node-bdfcs","kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"]
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.155301    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.156348    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.157511    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.157531    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.157564    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.157643    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fkf6b\""
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.158399    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.158578    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.158696    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.158752    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.158933    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zktt5\""
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.159771    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.159840    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.165928    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.165966    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166019    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166082    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166162    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166183    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.166195    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166293    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166303    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166329    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166353    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 10 18:58:18.234109 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166362    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4f9c2\""
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.166367    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166604    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166604    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166636    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8lpnd\""
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.166684    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168014    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-sysctl-conf\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168032    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-var-lib-kubelet\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168047    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-var-lib-kubelet\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-run-multus-certs\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168077    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-run\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168091    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w746\" (UniqueName: \"kubernetes.io/projected/389d4ea0-eee9-400b-a8c7-58ff953732d4-kube-api-access-2w746\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-sys\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168462    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168471    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-lib-modules\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ebfcae4d-1a86-4c05-b134-59ad0ec54c15-hosts-file\") pod \"node-resolver-4xtcp\" (UID: \"ebfcae4d-1a86-4c05-b134-59ad0ec54c15\") " pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168532    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168549    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95kbh\" (UniqueName: \"kubernetes.io/projected/ebfcae4d-1a86-4c05-b134-59ad0ec54c15-kube-api-access-95kbh\") pod \"node-resolver-4xtcp\" (UID: \"ebfcae4d-1a86-4c05-b134-59ad0ec54c15\") " pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168577    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-cni-binary-copy\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168609    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-daemon-config\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.236391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168639    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-cnibin\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168757    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jvf\" (UniqueName: \"kubernetes.io/projected/354e5926-7524-4d50-a294-6d30003bc759-kube-api-access-f5jvf\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168791    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-sysconfig\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168804    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168812    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vcrlz\""
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168817    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/389d4ea0-eee9-400b-a8c7-58ff953732d4-tmp\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168826    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-57wft\""
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168852    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-cni-dir\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168884    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-sysctl-d\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168963    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-cnibin\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.168994    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-run-netns\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169039    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-var-lib-cni-bin\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169071    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-var-lib-cni-multus\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169113    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-system-cni-dir\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169217    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-modprobe-d\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169271    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-kubernetes\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.237480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169311    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-systemd\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169348    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-host\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169372    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-tuned\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-system-cni-dir\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169437    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-os-release\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169491    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-socket-dir-parent\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169530    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebfcae4d-1a86-4c05-b134-59ad0ec54c15-tmp-dir\") pod \"node-resolver-4xtcp\" (UID: \"ebfcae4d-1a86-4c05-b134-59ad0ec54c15\") " pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169569    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-hostroot\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169609    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-conf-dir\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169646    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-etc-kubernetes\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169683    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/354e5926-7524-4d50-a294-6d30003bc759-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169721    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-run-k8s-cni-cncf-io\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169758    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2k72\" (UniqueName: \"kubernetes.io/projected/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-kube-api-access-t2k72\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169788    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-os-release\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169855    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/354e5926-7524-4d50-a294-6d30003bc759-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/354e5926-7524-4d50-a294-6d30003bc759-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.169937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2mz\" (UniqueName: \"kubernetes.io/projected/c553a64a-aa13-4a33-90c3-aee6b4f75207-kube-api-access-jm2mz\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.170929    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 10 18:58:18.239219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.170965    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.170897    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.170990    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.171083    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.170931    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.171367    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.171398    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.171409    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.171491    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bqfv5\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.171510    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.171528    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.171604    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4sq5b\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.171622    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.171668    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.172058    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.172130    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gh59\""
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.174103    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.194727    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-px5fq"
Jul 10 18:58:18.240246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.213596    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-px5fq"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.262109    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270116    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-conf-dir\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270137    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3bbdd3e9-01c4-4253-8fec-d519904e4e08-ovnkube-script-lib\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-socket-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270173    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-hostroot\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270189    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-etc-kubernetes\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270238    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-conf-dir\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270247    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-etc-kubernetes\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270290    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/354e5926-7524-4d50-a294-6d30003bc759-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270326    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-node-log\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270355    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2k72\" (UniqueName: \"kubernetes.io/projected/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-kube-api-access-t2k72\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270293    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-hostroot\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270393    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/354e5926-7524-4d50-a294-6d30003bc759-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270446    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-var-lib-openvswitch\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270469    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g77kf\" (UniqueName: \"kubernetes.io/projected/3bbdd3e9-01c4-4253-8fec-d519904e4e08-kube-api-access-g77kf\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270494    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-var-lib-kubelet\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270518    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aab81ea0-5cd4-4e18-b48e-70b3a18292b2-iptables-alerter-script\") pod \"iptables-alerter-72vr9\" (UID: \"aab81ea0-5cd4-4e18-b48e-70b3a18292b2\") " pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.343517 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270555    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aab81ea0-5cd4-4e18-b48e-70b3a18292b2-host-slash\") pod \"iptables-alerter-72vr9\" (UID: \"aab81ea0-5cd4-4e18-b48e-70b3a18292b2\") " pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270565    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-var-lib-kubelet\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270587    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270616    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w746\" (UniqueName: \"kubernetes.io/projected/389d4ea0-eee9-400b-a8c7-58ff953732d4-kube-api-access-2w746\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270636    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b8009c7-2984-43f5-8195-8c13750d2f2b-konnectivity-ca\") pod \"konnectivity-agent-64zsj\" (UID: \"7b8009c7-2984-43f5-8195-8c13750d2f2b\") " pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270679    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-kubelet-config\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270719    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-slash\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270758    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95kbh\" (UniqueName: \"kubernetes.io/projected/ebfcae4d-1a86-4c05-b134-59ad0ec54c15-kube-api-access-95kbh\") pod \"node-resolver-4xtcp\" (UID: \"ebfcae4d-1a86-4c05-b134-59ad0ec54c15\") " pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270778    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-cni-binary-copy\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270793    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-cnibin\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270808    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jvf\" (UniqueName: \"kubernetes.io/projected/354e5926-7524-4d50-a294-6d30003bc759-kube-api-access-f5jvf\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270836    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-sys-fs\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270841    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-cnibin\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270851    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-sysconfig\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270884    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-sysconfig\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270882    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/389d4ea0-eee9-400b-a8c7-58ff953732d4-tmp\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.346779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270946    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-run-systemd\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270978    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-cnibin\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.270992    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-run-netns\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271006    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-var-lib-cni-bin\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271026    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-var-lib-cni-multus\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271056    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-var-lib-cni-bin\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271049    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-cnibin\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271065    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-system-cni-dir\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271084    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-var-lib-cni-multus\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271098    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-system-cni-dir\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271055    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-run-netns\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-run-netns\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271138    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54-host\") pod \"node-ca-m9r7c\" (UID: \"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54\") " pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271154    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-modprobe-d\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271174    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-kubernetes\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271229    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-kubernetes\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271261    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-os-release\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.347882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271283    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-modprobe-d\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271298    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-os-release\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271307    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8tpw\" (UniqueName: \"kubernetes.io/projected/41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54-kube-api-access-k8tpw\") pod \"node-ca-m9r7c\" (UID: \"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54\") " pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-cni-netd\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271367    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bbdd3e9-01c4-4253-8fec-d519904e4e08-ovn-node-metrics-cert\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271383    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-socket-dir-parent\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271431    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebfcae4d-1a86-4c05-b134-59ad0ec54c15-tmp-dir\") pod \"node-resolver-4xtcp\" (UID: \"ebfcae4d-1a86-4c05-b134-59ad0ec54c15\") " pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271448    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-systemd-units\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271463    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-etc-openvswitch\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271484    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wc4k\" (UniqueName: \"kubernetes.io/projected/aab81ea0-5cd4-4e18-b48e-70b3a18292b2-kube-api-access-7wc4k\") pod \"iptables-alerter-72vr9\" (UID: \"aab81ea0-5cd4-4e18-b48e-70b3a18292b2\") " pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271507    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qr6r\" (UniqueName: \"kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r\") pod \"network-check-target-mgkkh\" (UID: \"76040466-2342-4008-becd-8b68f1f99e5e\") " pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271528    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-socket-dir-parent\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271555    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-run-k8s-cni-cncf-io\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271608    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-os-release\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/354e5926-7524-4d50-a294-6d30003bc759-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271657    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2mz\" (UniqueName: \"kubernetes.io/projected/c553a64a-aa13-4a33-90c3-aee6b4f75207-kube-api-access-jm2mz\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:18.350417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271680    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-os-release\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271673    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-run-k8s-cni-cncf-io\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271696    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-device-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271794    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-sysctl-conf\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271830    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-var-lib-kubelet\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271862    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-run-multus-certs\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54-serviceca\") pod \"node-ca-m9r7c\" (UID: \"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54\") " pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271944    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-run-openvswitch\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271961    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-sysctl-conf\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271966    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-run-multus-certs\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271972    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3bbdd3e9-01c4-4253-8fec-d519904e4e08-env-overrides\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.271995    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272018    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-host-var-lib-kubelet\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-run\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272049    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-sys\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272054    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-run\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272072    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-lib-modules\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.351301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ebfcae4d-1a86-4c05-b134-59ad0ec54c15-hosts-file\") pod \"node-resolver-4xtcp\" (UID: \"ebfcae4d-1a86-4c05-b134-59ad0ec54c15\") " pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-daemon-config\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272115    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-sys\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272134    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-run-ovn\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272163    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-lib-modules\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272166    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-log-socket\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272177    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ebfcae4d-1a86-4c05-b134-59ad0ec54c15-hosts-file\") pod \"node-resolver-4xtcp\" (UID: \"ebfcae4d-1a86-4c05-b134-59ad0ec54c15\") " pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272198    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-cni-bin\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272201    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272325    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-cni-dir\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272373    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272377    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/354e5926-7524-4d50-a294-6d30003bc759-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272393    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b8009c7-2984-43f5-8195-8c13750d2f2b-agent-certs\") pod \"konnectivity-agent-64zsj\" (UID: \"7b8009c7-2984-43f5-8195-8c13750d2f2b\") " pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272414    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-kubelet\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3bbdd3e9-01c4-4253-8fec-d519904e4e08-ovnkube-config\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-dbus\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:18.353479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272460    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-cni-dir\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6vk\" (UniqueName: \"kubernetes.io/projected/d84dd414-596c-467c-a2f2-e20ba666bf51-kube-api-access-pz6vk\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272516    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-sysctl-d\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.272542    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272543    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-registration-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272601    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-systemd\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272627    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-sysctl-d\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.272655    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs podName:c553a64a-aa13-4a33-90c3-aee6b4f75207 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:18.772613957 +0000 UTC m=+2.024261267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs") pod "network-metrics-daemon-cbt5f" (UID: "c553a64a-aa13-4a33-90c3-aee6b4f75207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272686    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-systemd\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272725    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-host\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-tuned\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272781    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-system-cni-dir\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272808    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/389d4ea0-eee9-400b-a8c7-58ff953732d4-host\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272844    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/354e5926-7524-4d50-a294-6d30003bc759-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272865    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-system-cni-dir\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272952    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebfcae4d-1a86-4c05-b134-59ad0ec54c15-tmp-dir\") pod \"node-resolver-4xtcp\" (UID: \"ebfcae4d-1a86-4c05-b134-59ad0ec54c15\") " pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.272963    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/354e5926-7524-4d50-a294-6d30003bc759-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.354347 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.273047    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/354e5926-7524-4d50-a294-6d30003bc759-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.357436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.273256    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-multus-daemon-config\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.357436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.273611    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-cni-binary-copy\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.357436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.275200    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/389d4ea0-eee9-400b-a8c7-58ff953732d4-etc-tuned\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.357436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.275227    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/389d4ea0-eee9-400b-a8c7-58ff953732d4-tmp\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.357436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.284962    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w746\" (UniqueName: \"kubernetes.io/projected/389d4ea0-eee9-400b-a8c7-58ff953732d4-kube-api-access-2w746\") pod \"tuned-6wg9f\" (UID: \"389d4ea0-eee9-400b-a8c7-58ff953732d4\") " pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.357436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.285045    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jvf\" (UniqueName: \"kubernetes.io/projected/354e5926-7524-4d50-a294-6d30003bc759-kube-api-access-f5jvf\") pod \"multus-additional-cni-plugins-8m2j6\" (UID: \"354e5926-7524-4d50-a294-6d30003bc759\") " pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.357436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.285058    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2mz\" (UniqueName: \"kubernetes.io/projected/c553a64a-aa13-4a33-90c3-aee6b4f75207-kube-api-access-jm2mz\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:18.357436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.285201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95kbh\" (UniqueName: \"kubernetes.io/projected/ebfcae4d-1a86-4c05-b134-59ad0ec54c15-kube-api-access-95kbh\") pod \"node-resolver-4xtcp\" (UID: \"ebfcae4d-1a86-4c05-b134-59ad0ec54c15\") " pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.357436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.285201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2k72\" (UniqueName: \"kubernetes.io/projected/cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4-kube-api-access-t2k72\") pod \"multus-5hnlc\" (UID: \"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4\") " pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.357436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.295390    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8m2j6"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373173    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373201    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-slash\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373238    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-sys-fs\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373278    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-sys-fs\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373278    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-slash\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373352    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-run-systemd\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-run-systemd\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373443    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-run-netns\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373468    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-run-netns\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373469    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54-host\") pod \"node-ca-m9r7c\" (UID: \"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54\") " pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373492    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8tpw\" (UniqueName: \"kubernetes.io/projected/41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54-kube-api-access-k8tpw\") pod \"node-ca-m9r7c\" (UID: \"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54\") " pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373504    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54-host\") pod \"node-ca-m9r7c\" (UID: \"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54\") " pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373516    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373553    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-cni-netd\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373576    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bbdd3e9-01c4-4253-8fec-d519904e4e08-ovn-node-metrics-cert\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.473463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373580    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373612    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-cni-netd\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-systemd-units\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373640    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-etc-openvswitch\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373655    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wc4k\" (UniqueName: \"kubernetes.io/projected/aab81ea0-5cd4-4e18-b48e-70b3a18292b2-kube-api-access-7wc4k\") pod \"iptables-alerter-72vr9\" (UID: \"aab81ea0-5cd4-4e18-b48e-70b3a18292b2\") " pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr6r\" (UniqueName: \"kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r\") pod \"network-check-target-mgkkh\" (UID: \"76040466-2342-4008-becd-8b68f1f99e5e\") " pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373706    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-etc-openvswitch\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373709    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-device-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373740    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-device-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373710    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-systemd-units\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54-serviceca\") pod \"node-ca-m9r7c\" (UID: \"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54\") " pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-run-openvswitch\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373813    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3bbdd3e9-01c4-4253-8fec-d519904e4e08-env-overrides\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-run-openvswitch\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373890    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.373902    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-run-ovn\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.475866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374003    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-log-socket\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-cni-bin\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374007    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-run-ovn\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374065    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374094    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-cni-bin\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374067    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-log-socket\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374110    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b8009c7-2984-43f5-8195-8c13750d2f2b-agent-certs\") pod \"konnectivity-agent-64zsj\" (UID: \"7b8009c7-2984-43f5-8195-8c13750d2f2b\") " pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374104    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374134    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-kubelet\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-host-kubelet\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3bbdd3e9-01c4-4253-8fec-d519904e4e08-ovnkube-config\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374237    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-dbus\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374261    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6vk\" (UniqueName: \"kubernetes.io/projected/d84dd414-596c-467c-a2f2-e20ba666bf51-kube-api-access-pz6vk\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374285    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-registration-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3bbdd3e9-01c4-4253-8fec-d519904e4e08-ovnkube-script-lib\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374336    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-socket-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374344    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3bbdd3e9-01c4-4253-8fec-d519904e4e08-env-overrides\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.476690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374374    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-node-log\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374381    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54-serviceca\") pod \"node-ca-m9r7c\" (UID: \"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54\") " pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-registration-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-var-lib-openvswitch\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374482    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-node-log\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374503    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g77kf\" (UniqueName: \"kubernetes.io/projected/3bbdd3e9-01c4-4253-8fec-d519904e4e08-kube-api-access-g77kf\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374556    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aab81ea0-5cd4-4e18-b48e-70b3a18292b2-iptables-alerter-script\") pod \"iptables-alerter-72vr9\" (UID: \"aab81ea0-5cd4-4e18-b48e-70b3a18292b2\") " pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d84dd414-596c-467c-a2f2-e20ba666bf51-socket-dir\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374580    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aab81ea0-5cd4-4e18-b48e-70b3a18292b2-host-slash\") pod \"iptables-alerter-72vr9\" (UID: \"aab81ea0-5cd4-4e18-b48e-70b3a18292b2\") " pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374608    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-dbus\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374630    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aab81ea0-5cd4-4e18-b48e-70b3a18292b2-host-slash\") pod \"iptables-alerter-72vr9\" (UID: \"aab81ea0-5cd4-4e18-b48e-70b3a18292b2\") " pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374748    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bbdd3e9-01c4-4253-8fec-d519904e4e08-var-lib-openvswitch\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374786    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3bbdd3e9-01c4-4253-8fec-d519904e4e08-ovnkube-config\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374785    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b8009c7-2984-43f5-8195-8c13750d2f2b-konnectivity-ca\") pod \"konnectivity-agent-64zsj\" (UID: \"7b8009c7-2984-43f5-8195-8c13750d2f2b\") " pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.374863    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374883    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-kubelet-config\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:18.612544 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.374954    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret podName:d68b9eae-5b48-4dfa-bfd7-185a928fabd7 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:18.874935027 +0000 UTC m=+2.126582338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret") pod "global-pull-secret-syncer-4xprr" (UID: "d68b9eae-5b48-4dfa-bfd7-185a928fabd7") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.374965    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-kubelet-config\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.375171    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3bbdd3e9-01c4-4253-8fec-d519904e4e08-ovnkube-script-lib\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.375501    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b8009c7-2984-43f5-8195-8c13750d2f2b-konnectivity-ca\") pod \"konnectivity-agent-64zsj\" (UID: \"7b8009c7-2984-43f5-8195-8c13750d2f2b\") " pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.375530    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aab81ea0-5cd4-4e18-b48e-70b3a18292b2-iptables-alerter-script\") pod \"iptables-alerter-72vr9\" (UID: \"aab81ea0-5cd4-4e18-b48e-70b3a18292b2\") " pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.375818    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bbdd3e9-01c4-4253-8fec-d519904e4e08-ovn-node-metrics-cert\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.376762    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b8009c7-2984-43f5-8195-8c13750d2f2b-agent-certs\") pod \"konnectivity-agent-64zsj\" (UID: \"7b8009c7-2984-43f5-8195-8c13750d2f2b\") " pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.379726    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.379759    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.379776    2567 projected.go:196] Error preparing data for projected volume kube-api-access-7qr6r for pod openshift-network-diagnostics/network-check-target-mgkkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.379841    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r podName:76040466-2342-4008-becd-8b68f1f99e5e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:18.879824769 +0000 UTC m=+2.131472093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7qr6r" (UniqueName: "kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r") pod "network-check-target-mgkkh" (UID: "76040466-2342-4008-becd-8b68f1f99e5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.381405    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6vk\" (UniqueName: \"kubernetes.io/projected/d84dd414-596c-467c-a2f2-e20ba666bf51-kube-api-access-pz6vk\") pod \"aws-ebs-csi-driver-node-dv7zt\" (UID: \"d84dd414-596c-467c-a2f2-e20ba666bf51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.381405    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wc4k\" (UniqueName: \"kubernetes.io/projected/aab81ea0-5cd4-4e18-b48e-70b3a18292b2-kube-api-access-7wc4k\") pod \"iptables-alerter-72vr9\" (UID: \"aab81ea0-5cd4-4e18-b48e-70b3a18292b2\") " pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.381726    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8tpw\" (UniqueName: \"kubernetes.io/projected/41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54-kube-api-access-k8tpw\") pod \"node-ca-m9r7c\" (UID: \"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54\") " pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.382846    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g77kf\" (UniqueName: \"kubernetes.io/projected/3bbdd3e9-01c4-4253-8fec-d519904e4e08-kube-api-access-g77kf\") pod \"ovnkube-node-bdfcs\" (UID: \"3bbdd3e9-01c4-4253-8fec-d519904e4e08\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.396454    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.401298    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.490820    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6wg9f"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.502973    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4xtcp"
Jul 10 18:58:18.614551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.510747    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5hnlc"
Jul 10 18:58:18.615643 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.549315    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18d246e0bedf5b391d12d0230022e47.slice/crio-3cb371d369d56a2ce171eebcc44a858b45470a28e59f15dbaa9c54df337f53ff WatchSource:0}: Error finding container 3cb371d369d56a2ce171eebcc44a858b45470a28e59f15dbaa9c54df337f53ff: Status 404 returned error can't find the container with id 3cb371d369d56a2ce171eebcc44a858b45470a28e59f15dbaa9c54df337f53ff
Jul 10 18:58:18.615643 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.549505    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b86e7df3f891ef5281cf18ca965a9f5.slice/crio-2a5a9ca301c9340546ceef13664fd1dcd7ca8dbc85567c7647e7a988b9d7ffb2 WatchSource:0}: Error finding container 2a5a9ca301c9340546ceef13664fd1dcd7ca8dbc85567c7647e7a988b9d7ffb2: Status 404 returned error can't find the container with id 2a5a9ca301c9340546ceef13664fd1dcd7ca8dbc85567c7647e7a988b9d7ffb2
Jul 10 18:58:18.615643 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.557089    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:58:18.615643 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.602605    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:18.615643 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.613267    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8009c7_2984_43f5_8195_8c13750d2f2b.slice/crio-8231846a4651357db3b1a852c93c3641624e48e1af94a7fca31105c74bf21f3e WatchSource:0}: Error finding container 8231846a4651357db3b1a852c93c3641624e48e1af94a7fca31105c74bf21f3e: Status 404 returned error can't find the container with id 8231846a4651357db3b1a852c93c3641624e48e1af94a7fca31105c74bf21f3e
Jul 10 18:58:18.632396 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.632377    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-72vr9"
Jul 10 18:58:18.640771 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.640741    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab81ea0_5cd4_4e18_b48e_70b3a18292b2.slice/crio-de9820184a15ebc3036740229478572001fbb7c4545a046be039664691291875 WatchSource:0}: Error finding container de9820184a15ebc3036740229478572001fbb7c4545a046be039664691291875: Status 404 returned error can't find the container with id de9820184a15ebc3036740229478572001fbb7c4545a046be039664691291875
Jul 10 18:58:18.676931 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.676901    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m9r7c"
Jul 10 18:58:18.694005 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.693975    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ebb4c5_aab3_47f9_8c72_9c7d4ffe1a54.slice/crio-2cad6eb3543337888e70dc848b9ccd3d10469c32b0791a6e9a3c407078a1f8df WatchSource:0}: Error finding container 2cad6eb3543337888e70dc848b9ccd3d10469c32b0791a6e9a3c407078a1f8df: Status 404 returned error can't find the container with id 2cad6eb3543337888e70dc848b9ccd3d10469c32b0791a6e9a3c407078a1f8df
Jul 10 18:58:18.724096 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.724068    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354e5926_7524_4d50_a294_6d30003bc759.slice/crio-12094408b7e255cebe38e8cc69d041061c2612c6ea4e803b8d766034e5e89a0b WatchSource:0}: Error finding container 12094408b7e255cebe38e8cc69d041061c2612c6ea4e803b8d766034e5e89a0b: Status 404 returned error can't find the container with id 12094408b7e255cebe38e8cc69d041061c2612c6ea4e803b8d766034e5e89a0b
Jul 10 18:58:18.730087 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.730069    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84dd414_596c_467c_a2f2_e20ba666bf51.slice/crio-b6b555b80b3e1b18837d912cb73d4628b3a072ab25c1d7a7160f9d00d46b4bd9 WatchSource:0}: Error finding container b6b555b80b3e1b18837d912cb73d4628b3a072ab25c1d7a7160f9d00d46b4bd9: Status 404 returned error can't find the container with id b6b555b80b3e1b18837d912cb73d4628b3a072ab25c1d7a7160f9d00d46b4bd9
Jul 10 18:58:18.735954 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.735936    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbdd3e9_01c4_4253_8fec_d519904e4e08.slice/crio-c639c453390675415427e4b610623318144e976e9a3160968fbf6e476e672ef6 WatchSource:0}: Error finding container c639c453390675415427e4b610623318144e976e9a3160968fbf6e476e672ef6: Status 404 returned error can't find the container with id c639c453390675415427e4b610623318144e976e9a3160968fbf6e476e672ef6
Jul 10 18:58:18.782490 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.782469    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:18.782608 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.782594    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:18.782646 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.782641    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs podName:c553a64a-aa13-4a33-90c3-aee6b4f75207 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:19.78262682 +0000 UTC m=+3.034274130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs") pod "network-metrics-daemon-cbt5f" (UID: "c553a64a-aa13-4a33-90c3-aee6b4f75207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:18.792423 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.792404    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfcae4d_1a86_4c05_b134_59ad0ec54c15.slice/crio-7d97b6845b703e92a6d920238417c2481ffdb461f3ad65d0961c6c9223844b02 WatchSource:0}: Error finding container 7d97b6845b703e92a6d920238417c2481ffdb461f3ad65d0961c6c9223844b02: Status 404 returned error can't find the container with id 7d97b6845b703e92a6d920238417c2481ffdb461f3ad65d0961c6c9223844b02
Jul 10 18:58:18.809441 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.809419    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389d4ea0_eee9_400b_a8c7_58ff953732d4.slice/crio-5d2f170f39943fcc6ce9408f2081d1adf98d5135d0cb29c6c9a28a3094b239cf WatchSource:0}: Error finding container 5d2f170f39943fcc6ce9408f2081d1adf98d5135d0cb29c6c9a28a3094b239cf: Status 404 returned error can't find the container with id 5d2f170f39943fcc6ce9408f2081d1adf98d5135d0cb29c6c9a28a3094b239cf
Jul 10 18:58:18.840033 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:18.840013    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd3893d_7ab5_4d0c_b7e4_39a0e66ab1c4.slice/crio-9c24d80b7d26fae021bb579987669455eb0934d7bae129c37772625b571ff887 WatchSource:0}: Error finding container 9c24d80b7d26fae021bb579987669455eb0934d7bae129c37772625b571ff887: Status 404 returned error can't find the container with id 9c24d80b7d26fae021bb579987669455eb0934d7bae129c37772625b571ff887
Jul 10 18:58:18.883194 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.883147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:18.883282 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.883196    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr6r\" (UniqueName: \"kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r\") pod \"network-check-target-mgkkh\" (UID: \"76040466-2342-4008-becd-8b68f1f99e5e\") " pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:18.883344 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.883285    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:18.883344 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.883332    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret podName:d68b9eae-5b48-4dfa-bfd7-185a928fabd7 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:19.883317722 +0000 UTC m=+3.134965033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret") pod "global-pull-secret-syncer-4xprr" (UID: "d68b9eae-5b48-4dfa-bfd7-185a928fabd7") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:18.883344 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.883337    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:18.883476 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.883355    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:18.883476 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.883368    2567 projected.go:196] Error preparing data for projected volume kube-api-access-7qr6r for pod openshift-network-diagnostics/network-check-target-mgkkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:18.883476 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:18.883420    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r podName:76040466-2342-4008-becd-8b68f1f99e5e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:19.88340379 +0000 UTC m=+3.135051098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qr6r" (UniqueName: "kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r") pod "network-check-target-mgkkh" (UID: "76040466-2342-4008-becd-8b68f1f99e5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:18.981403 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:18.981373    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:19.215554 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.215445    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:53:18 +0000 UTC" deadline="2028-04-04 03:22:43.674110744 +0000 UTC"
Jul 10 18:58:19.215554 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.215506    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15200h24m24.458608817s"
Jul 10 18:58:19.413562 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.413533    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:19.422728 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.422660    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:19.439324 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.439302    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:19.439430 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:19.439412    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:19.457394 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.457338    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-64zsj" event={"ID":"7b8009c7-2984-43f5-8195-8c13750d2f2b","Type":"ContainerStarted","Data":"8231846a4651357db3b1a852c93c3641624e48e1af94a7fca31105c74bf21f3e"}
Jul 10 18:58:19.466362 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.466268    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal" event={"ID":"3b86e7df3f891ef5281cf18ca965a9f5","Type":"ContainerStarted","Data":"2a5a9ca301c9340546ceef13664fd1dcd7ca8dbc85567c7647e7a988b9d7ffb2"}
Jul 10 18:58:19.473547 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.473517    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs" event={"ID":"3bbdd3e9-01c4-4253-8fec-d519904e4e08","Type":"ContainerStarted","Data":"c639c453390675415427e4b610623318144e976e9a3160968fbf6e476e672ef6"}
Jul 10 18:58:19.487080 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.487007    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt" event={"ID":"d84dd414-596c-467c-a2f2-e20ba666bf51","Type":"ContainerStarted","Data":"b6b555b80b3e1b18837d912cb73d4628b3a072ab25c1d7a7160f9d00d46b4bd9"}
Jul 10 18:58:19.494308 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.494280    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m2j6" event={"ID":"354e5926-7524-4d50-a294-6d30003bc759","Type":"ContainerStarted","Data":"12094408b7e255cebe38e8cc69d041061c2612c6ea4e803b8d766034e5e89a0b"}
Jul 10 18:58:19.511112 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.511047    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m9r7c" event={"ID":"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54","Type":"ContainerStarted","Data":"2cad6eb3543337888e70dc848b9ccd3d10469c32b0791a6e9a3c407078a1f8df"}
Jul 10 18:58:19.521578 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.521552    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal" event={"ID":"d18d246e0bedf5b391d12d0230022e47","Type":"ContainerStarted","Data":"3cb371d369d56a2ce171eebcc44a858b45470a28e59f15dbaa9c54df337f53ff"}
Jul 10 18:58:19.533710 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.533685    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5hnlc" event={"ID":"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4","Type":"ContainerStarted","Data":"9c24d80b7d26fae021bb579987669455eb0934d7bae129c37772625b571ff887"}
Jul 10 18:58:19.548668 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.548590    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6wg9f" event={"ID":"389d4ea0-eee9-400b-a8c7-58ff953732d4","Type":"ContainerStarted","Data":"5d2f170f39943fcc6ce9408f2081d1adf98d5135d0cb29c6c9a28a3094b239cf"}
Jul 10 18:58:19.567221 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.567187    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4xtcp" event={"ID":"ebfcae4d-1a86-4c05-b134-59ad0ec54c15","Type":"ContainerStarted","Data":"7d97b6845b703e92a6d920238417c2481ffdb461f3ad65d0961c6c9223844b02"}
Jul 10 18:58:19.577525 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.577499    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-72vr9" event={"ID":"aab81ea0-5cd4-4e18-b48e-70b3a18292b2","Type":"ContainerStarted","Data":"de9820184a15ebc3036740229478572001fbb7c4545a046be039664691291875"}
Jul 10 18:58:19.792598 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.792048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:19.792598 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:19.792178    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:19.792598 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:19.792236    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs podName:c553a64a-aa13-4a33-90c3-aee6b4f75207 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:21.792217866 +0000 UTC m=+5.043865177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs") pod "network-metrics-daemon-cbt5f" (UID: "c553a64a-aa13-4a33-90c3-aee6b4f75207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:19.892847 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.892306    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:19.892847 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:19.892349    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr6r\" (UniqueName: \"kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r\") pod \"network-check-target-mgkkh\" (UID: \"76040466-2342-4008-becd-8b68f1f99e5e\") " pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:19.892847 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:19.892464    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:19.892847 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:19.892478    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:19.892847 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:19.892487    2567 projected.go:196] Error preparing data for projected volume kube-api-access-7qr6r for pod openshift-network-diagnostics/network-check-target-mgkkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:19.892847 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:19.892527    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r podName:76040466-2342-4008-becd-8b68f1f99e5e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:21.892513989 +0000 UTC m=+5.144161295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qr6r" (UniqueName: "kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r") pod "network-check-target-mgkkh" (UID: "76040466-2342-4008-becd-8b68f1f99e5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:19.892847 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:19.892792    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:19.892847 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:19.892820    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret podName:d68b9eae-5b48-4dfa-bfd7-185a928fabd7 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:21.892811627 +0000 UTC m=+5.144458935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret") pod "global-pull-secret-syncer-4xprr" (UID: "d68b9eae-5b48-4dfa-bfd7-185a928fabd7") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:20.216289 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:20.216250    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:53:18 +0000 UTC" deadline="2028-03-12 15:40:44.277339227 +0000 UTC"
Jul 10 18:58:20.216289 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:20.216289    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14660h42m24.061055183s"
Jul 10 18:58:20.414012 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:20.413945    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:20.414187 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:20.414070    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:20.414436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:20.414274    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:20.414436 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:20.414397    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:21.414932 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:21.414199    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:21.414932 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:21.414327    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:21.807998 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:21.807952    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:21.808171 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:21.808113    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:21.808234 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:21.808172    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs podName:c553a64a-aa13-4a33-90c3-aee6b4f75207 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:25.80815316 +0000 UTC m=+9.059800470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs") pod "network-metrics-daemon-cbt5f" (UID: "c553a64a-aa13-4a33-90c3-aee6b4f75207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:21.908987 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:21.908954    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr6r\" (UniqueName: \"kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r\") pod \"network-check-target-mgkkh\" (UID: \"76040466-2342-4008-becd-8b68f1f99e5e\") " pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:21.909155 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:21.909040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:21.909218 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:21.909166    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:21.909268 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:21.909219    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret podName:d68b9eae-5b48-4dfa-bfd7-185a928fabd7 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:25.909200813 +0000 UTC m=+9.160848125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret") pod "global-pull-secret-syncer-4xprr" (UID: "d68b9eae-5b48-4dfa-bfd7-185a928fabd7") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:21.909608 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:21.909588    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:21.909682 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:21.909617    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:21.909682 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:21.909630    2567 projected.go:196] Error preparing data for projected volume kube-api-access-7qr6r for pod openshift-network-diagnostics/network-check-target-mgkkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:21.909682 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:21.909674    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r podName:76040466-2342-4008-becd-8b68f1f99e5e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:25.909659695 +0000 UTC m=+9.161307005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qr6r" (UniqueName: "kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r") pod "network-check-target-mgkkh" (UID: "76040466-2342-4008-becd-8b68f1f99e5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:22.413671 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:22.413633    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:22.413860 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:22.413770    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:22.413968 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:22.413946    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:22.414102 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:22.414078    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:23.413720 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:23.413655    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:23.414226 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:23.413837    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:24.413741 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:24.413706    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:24.414222 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:24.413864    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:24.414222 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:24.413935    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:24.414222 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:24.414054    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:25.414543 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:25.414000    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:25.414543 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:25.414129    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:25.840101 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:25.840062    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:25.840296 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:25.840225    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:25.840362 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:25.840301    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs podName:c553a64a-aa13-4a33-90c3-aee6b4f75207 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:33.840278259 +0000 UTC m=+17.091925575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs") pod "network-metrics-daemon-cbt5f" (UID: "c553a64a-aa13-4a33-90c3-aee6b4f75207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:25.942052 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:25.941214    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr6r\" (UniqueName: \"kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r\") pod \"network-check-target-mgkkh\" (UID: \"76040466-2342-4008-becd-8b68f1f99e5e\") " pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:25.942052 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:25.941309    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:25.942052 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:25.941446    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:25.942052 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:25.941510    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret podName:d68b9eae-5b48-4dfa-bfd7-185a928fabd7 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:33.941491038 +0000 UTC m=+17.193138348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret") pod "global-pull-secret-syncer-4xprr" (UID: "d68b9eae-5b48-4dfa-bfd7-185a928fabd7") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:25.942052 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:25.941924    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:25.942052 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:25.941945    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:25.942052 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:25.941959    2567 projected.go:196] Error preparing data for projected volume kube-api-access-7qr6r for pod openshift-network-diagnostics/network-check-target-mgkkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:25.942052 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:25.942006    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r podName:76040466-2342-4008-becd-8b68f1f99e5e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:33.941990188 +0000 UTC m=+17.193637509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qr6r" (UniqueName: "kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r") pod "network-check-target-mgkkh" (UID: "76040466-2342-4008-becd-8b68f1f99e5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:26.413773 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:26.413741    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:26.413946 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:26.413865    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:26.414284 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:26.414211    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:26.414433 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:26.414322    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:27.414092 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:27.414060    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:27.414557 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:27.414179    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:28.413233 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:28.413193    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:28.413403 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:28.413193    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:28.413403 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:28.413364    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:28.413528 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:28.413440    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:29.413786 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:29.413760    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:29.414222 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:29.413882    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:30.413430 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:30.413373    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:30.413589 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:30.413385    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:30.413589 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:30.413529    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:30.413707 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:30.413598    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:31.414040 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:31.414004    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:31.414504 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:31.414111    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:32.414050 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:32.414018    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:32.414050 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:32.414046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:32.414493 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:32.414124    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:32.414493 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:32.414272    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:33.413528 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:33.413494    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:33.413707 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:33.413624    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:33.903775 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:33.903738    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:33.904220 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:33.903960    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:33.904220 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:33.904045    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs podName:c553a64a-aa13-4a33-90c3-aee6b4f75207 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:49.904021848 +0000 UTC m=+33.155669174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs") pod "network-metrics-daemon-cbt5f" (UID: "c553a64a-aa13-4a33-90c3-aee6b4f75207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:34.004744 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:34.004711    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr6r\" (UniqueName: \"kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r\") pod \"network-check-target-mgkkh\" (UID: \"76040466-2342-4008-becd-8b68f1f99e5e\") " pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:34.004932 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:34.004779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:34.004932 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:34.004872    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:34.004932 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:34.004887    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:34.004932 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:34.004930    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:34.005135 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:34.004944    2567 projected.go:196] Error preparing data for projected volume kube-api-access-7qr6r for pod openshift-network-diagnostics/network-check-target-mgkkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:34.005135 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:34.004948    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret podName:d68b9eae-5b48-4dfa-bfd7-185a928fabd7 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:50.004928346 +0000 UTC m=+33.256575657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret") pod "global-pull-secret-syncer-4xprr" (UID: "d68b9eae-5b48-4dfa-bfd7-185a928fabd7") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:34.005135 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:34.004991    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r podName:76040466-2342-4008-becd-8b68f1f99e5e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:50.004975577 +0000 UTC m=+33.256622902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qr6r" (UniqueName: "kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r") pod "network-check-target-mgkkh" (UID: "76040466-2342-4008-becd-8b68f1f99e5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:34.413283 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:34.413255    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:34.413417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:34.413256    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:34.413417 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:34.413377    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:34.413502 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:34.413430    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:35.413485 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:35.413454    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:35.413963 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:35.413568    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:36.413342 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:36.413302    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:36.413342 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:36.413331    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:36.413561 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:36.413416    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:36.413862 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:36.413559    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:37.414597 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:37.414562    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:37.415037 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:37.414679    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:38.413378 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:38.413358    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:38.413474 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:38.413389    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:38.413522 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:38.413472    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:38.413605 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:38.413586    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:39.413791 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.413632    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:39.414393 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:39.413862    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:39.626629 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.626549    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs" event={"ID":"3bbdd3e9-01c4-4253-8fec-d519904e4e08","Type":"ContainerStarted","Data":"f63eeac862d79de9fe53c17ef36e9408fdb15b92b31fabf2aad11173b8390fe9"}
Jul 10 18:58:39.626758 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.626641    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs" event={"ID":"3bbdd3e9-01c4-4253-8fec-d519904e4e08","Type":"ContainerStarted","Data":"4ce618b16cb3c23891c7b9bffd32fe3b654b0e317ee4b4d2314f465da88c8cfd"}
Jul 10 18:58:39.626758 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.626689    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs" event={"ID":"3bbdd3e9-01c4-4253-8fec-d519904e4e08","Type":"ContainerStarted","Data":"336dde503023703e6252d4b01abcc9b64946bd27929241c42afb345be587fe63"}
Jul 10 18:58:39.626758 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.626738    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs" event={"ID":"3bbdd3e9-01c4-4253-8fec-d519904e4e08","Type":"ContainerStarted","Data":"c7178bc1051085a6a1fd16868c2eb5724c888b8b46ef8d7d26f772df235099c8"}
Jul 10 18:58:39.626945 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.626783    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs" event={"ID":"3bbdd3e9-01c4-4253-8fec-d519904e4e08","Type":"ContainerStarted","Data":"2e812b8cc88e96d74304b3cfb8020a134df819328d9a24c91e2a82c62d39d5ea"}
Jul 10 18:58:39.626945 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.626829    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs" event={"ID":"3bbdd3e9-01c4-4253-8fec-d519904e4e08","Type":"ContainerStarted","Data":"270b6017958e3ff41113420f520de69f07aa3a9d575c5bab5afc21b00f1db81e"}
Jul 10 18:58:39.628319 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.628290    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt" event={"ID":"d84dd414-596c-467c-a2f2-e20ba666bf51","Type":"ContainerStarted","Data":"4437e4c01502f0d526a385801f52e9b8668e98f014ea13d50219c1d8f42a53e3"}
Jul 10 18:58:39.629923 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.629884    2567 generic.go:356] "Generic (PLEG): container finished" podID="354e5926-7524-4d50-a294-6d30003bc759" containerID="4815a4080385fdeb36da2f4d8e75d7d9a5f94eff4876d6de36e0ab0dc4abbf9d" exitCode=0
Jul 10 18:58:39.630028 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.629933    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m2j6" event={"ID":"354e5926-7524-4d50-a294-6d30003bc759","Type":"ContainerDied","Data":"4815a4080385fdeb36da2f4d8e75d7d9a5f94eff4876d6de36e0ab0dc4abbf9d"}
Jul 10 18:58:39.631596 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.631487    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m9r7c" event={"ID":"41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54","Type":"ContainerStarted","Data":"2cdbdc04cbc51d424e946ce1c145ce9d2ad6eaeb8ab9196d2e97ae2fbba033d4"}
Jul 10 18:58:39.633271 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.633248    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal" event={"ID":"d18d246e0bedf5b391d12d0230022e47","Type":"ContainerStarted","Data":"6022069295b12d380123524eb98439ea0d95e00e94d723a7a0a02d5fec709b25"}
Jul 10 18:58:39.634963 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.634941    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5hnlc" event={"ID":"cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4","Type":"ContainerStarted","Data":"d639f708de1afd26a37dcae40161105e7cf6b86d83a5beebd184af0c166f9d63"}
Jul 10 18:58:39.636633 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.636612    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6wg9f" event={"ID":"389d4ea0-eee9-400b-a8c7-58ff953732d4","Type":"ContainerStarted","Data":"e2b281d77ef0b9394a76c38c54b4f0ecdc91ee02c65b71c72199f80abcf081a7"}
Jul 10 18:58:39.638168 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.638143    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4xtcp" event={"ID":"ebfcae4d-1a86-4c05-b134-59ad0ec54c15","Type":"ContainerStarted","Data":"5f56775addc26bbe609072b57b25d438cf1c9146f5471a1216b51dd83ac13a18"}
Jul 10 18:58:39.639598 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.639573    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-64zsj" event={"ID":"7b8009c7-2984-43f5-8195-8c13750d2f2b","Type":"ContainerStarted","Data":"ca3c263bf02117142d75f1f50cb838f1cb75c6b585f43c06f4b1db8c11491256"}
Jul 10 18:58:39.641224 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.641201    2567 generic.go:356] "Generic (PLEG): container finished" podID="3b86e7df3f891ef5281cf18ca965a9f5" containerID="ef794a89c26e254e767d40d789f617b4847680d4e040baa0e2e9ad53590f202e" exitCode=0
Jul 10 18:58:39.641326 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.641235    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal" event={"ID":"3b86e7df3f891ef5281cf18ca965a9f5","Type":"ContainerDied","Data":"ef794a89c26e254e767d40d789f617b4847680d4e040baa0e2e9ad53590f202e"}
Jul 10 18:58:39.670438 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.670395    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6wg9f" podStartSLOduration=3.024622121 podStartE2EDuration="22.670379095s" totalImagesPullingTime="19.645756974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:39.669747286 +0000 UTC m=+22.921394612" watchObservedRunningTime="2026-07-10 18:58:39.670379095 +0000 UTC m=+22.922026425"
Jul 10 18:58:39.690187 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.690144    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5hnlc" podStartSLOduration=3.042277411 podStartE2EDuration="22.690127083s" totalImagesPullingTime="19.647849672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:39.68994578 +0000 UTC m=+22.941593113" watchObservedRunningTime="2026-07-10 18:58:39.690127083 +0000 UTC m=+22.941774415"
Jul 10 18:58:39.729557 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.729520    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-64zsj" podStartSLOduration=11.213977675 podStartE2EDuration="22.729503948s" totalImagesPullingTime="11.515526273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:39.708780251 +0000 UTC m=+22.960427578" watchObservedRunningTime="2026-07-10 18:58:39.729503948 +0000 UTC m=+22.981151337"
Jul 10 18:58:39.729649 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.729596    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m9r7c" podStartSLOduration=2.996567767 podStartE2EDuration="22.729586524s" totalImagesPullingTime="19.733018757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:39.728899388 +0000 UTC m=+22.980546717" watchObservedRunningTime="2026-07-10 18:58:39.729586524 +0000 UTC m=+22.981233853"
Jul 10 18:58:39.749862 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.749830    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4xtcp" podStartSLOduration=3.116357747 podStartE2EDuration="22.749816415s" totalImagesPullingTime="19.633458668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:39.749322132 +0000 UTC m=+23.000969461" watchObservedRunningTime="2026-07-10 18:58:39.749816415 +0000 UTC m=+23.001463743"
Jul 10 18:58:39.785680 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.785644    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-15.ec2.internal" podStartSLOduration=22.78563311 podStartE2EDuration="22.78563311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:39.785215952 +0000 UTC m=+23.036863281" watchObservedRunningTime="2026-07-10 18:58:39.78563311 +0000 UTC m=+23.037280438"
Jul 10 18:58:39.887610 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:39.887525    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:40.032314 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.032281    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:40.114285 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.114237    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:40.180755 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.180690    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:40.278154 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.278125    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:40.413662 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.413639    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:40.413771 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.413673    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:40.413771 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:40.413752    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:40.414146 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:40.413872    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:40.439223 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.439156    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 10 18:58:40.645942 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.645894    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt" event={"ID":"d84dd414-596c-467c-a2f2-e20ba666bf51","Type":"ContainerStarted","Data":"f00877f9088e68a112d92b77ebd224c31f24401bf1a3f0a71326d32720b548b8"}
Jul 10 18:58:40.647251 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.647227    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-72vr9" event={"ID":"aab81ea0-5cd4-4e18-b48e-70b3a18292b2","Type":"ContainerStarted","Data":"1db46980611a402acf9faf9898cfaed5124f00ddfbb6524e9a3d97599b5ea9d9"}
Jul 10 18:58:40.649298 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.649244    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal" event={"ID":"3b86e7df3f891ef5281cf18ca965a9f5","Type":"ContainerStarted","Data":"85ae189a2ae54a413ccbc1d2e4b2941c7ab6cc692b97357a83748f10be521d55"}
Jul 10 18:58:40.677455 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.677419    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-72vr9" podStartSLOduration=7.961545184 podStartE2EDuration="23.677401906s" totalImagesPullingTime="15.715856722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:40.662303402 +0000 UTC m=+23.913950730" watchObservedRunningTime="2026-07-10 18:58:40.677401906 +0000 UTC m=+23.929049236"
Jul 10 18:58:40.677611 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.677590    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-15.ec2.internal" podStartSLOduration=23.67758232 podStartE2EDuration="23.67758232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:40.677141836 +0000 UTC m=+23.928789165" watchObservedRunningTime="2026-07-10 18:58:40.67758232 +0000 UTC m=+23.929229651"
Jul 10 18:58:40.827743 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.827716    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:40.961348 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:40.961313    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:41.175678 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:41.175616    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:41.238609 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:41.238540    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:41.294621 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:41.294502    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:58:40.439176598Z","UUID":"3c098d12-e9d7-40c8-8dab-1dff7ec873cb","Handler":null,"Name":"","Endpoint":""}
Jul 10 18:58:41.296583 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:41.296539    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 10 18:58:41.296583 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:41.296581    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 10 18:58:41.413649 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:41.413616    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:41.413783 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:41.413737    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:41.973268 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:41.973166    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:42.284724 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.284622    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:42.285273 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.285236    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:42.361219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.361192    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:42.413313 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.413281    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:42.413443 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.413282    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:42.413443 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:42.413390    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:42.413564 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:42.413495    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:42.461263 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.461236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:42.656288 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.656248    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs" event={"ID":"3bbdd3e9-01c4-4253-8fec-d519904e4e08","Type":"ContainerStarted","Data":"8815c27d6c69051b3efd9ee520034c854fbb8c1138adc33a3941bb391deefbd4"}
Jul 10 18:58:42.658165 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.658139    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt" event={"ID":"d84dd414-596c-467c-a2f2-e20ba666bf51","Type":"ContainerStarted","Data":"38c60ef71125ff3ba7d870c5e3a1adca0a63f53004de7630b54a8495e0ae3711"}
Jul 10 18:58:42.658345 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.658332    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:42.658701 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.658687    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-64zsj"
Jul 10 18:58:42.677362 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.677326    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zt" podStartSLOduration=3.477736196 podStartE2EDuration="25.677312616s" totalImagesPullingTime="22.19957642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:42.677095682 +0000 UTC m=+25.928743012" watchObservedRunningTime="2026-07-10 18:58:42.677312616 +0000 UTC m=+25.928959945"
Jul 10 18:58:42.751083 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:42.751054    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:43.414207 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:43.414180    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:43.414722 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:43.414304    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:43.490866 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:43.490838    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:43.592099 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:43.592050    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:43.654151 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:43.654100    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:44.413545 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:44.413400    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:44.413656 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:44.413410    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:44.413656 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:44.413633    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:44.413763 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:44.413739    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:44.668087 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:44.668051    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs" event={"ID":"3bbdd3e9-01c4-4253-8fec-d519904e4e08","Type":"ContainerStarted","Data":"250dd90e8a8f1e90df298dc24f0552b731a0e821a827be148280db7114b63ada"}
Jul 10 18:58:44.668688 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:44.668288    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:44.668688 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:44.668346    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:44.670047 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:44.670024    2567 generic.go:356] "Generic (PLEG): container finished" podID="354e5926-7524-4d50-a294-6d30003bc759" containerID="11ec38bb52c6b95bbfa2f65c93a872df0b9e265e6ff7eb6371bdec180da3e4a2" exitCode=0
Jul 10 18:58:44.670158 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:44.670097    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m2j6" event={"ID":"354e5926-7524-4d50-a294-6d30003bc759","Type":"ContainerDied","Data":"11ec38bb52c6b95bbfa2f65c93a872df0b9e265e6ff7eb6371bdec180da3e4a2"}
Jul 10 18:58:44.683350 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:44.683333    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:44.699747 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:44.699712    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs" podStartSLOduration=7.597455104 podStartE2EDuration="27.699701328s" totalImagesPullingTime="20.102246224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:44.699557996 +0000 UTC m=+27.951205335" watchObservedRunningTime="2026-07-10 18:58:44.699701328 +0000 UTC m=+27.951348656"
Jul 10 18:58:45.253955 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:45.253921    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:45.308608 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:45.308568    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:45.414050 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:45.414022    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:45.414222 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:45.414148    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:45.615059 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:45.614979    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:45.673246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:45.673219    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:45.688009 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:45.687988    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:58:45.716404 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:45.716377    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:46.183642 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:46.183462    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mgkkh"]
Jul 10 18:58:46.183803 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:46.183795    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:46.183899 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:46.183875    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:46.186736 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:46.186667    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4xprr"]
Jul 10 18:58:46.186859 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:46.186785    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:46.186938 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:46.186864    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:46.187388 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:46.187362    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cbt5f"]
Jul 10 18:58:46.187493 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:46.187469    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:46.187589 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:46.187566    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:46.676570 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:46.676541    2567 generic.go:356] "Generic (PLEG): container finished" podID="354e5926-7524-4d50-a294-6d30003bc759" containerID="085ee8bdca92e9274ce5d90780a1a341b5e1ff30fbb0fc1eb4f052326e039065" exitCode=0
Jul 10 18:58:46.676924 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:46.676630    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m2j6" event={"ID":"354e5926-7524-4d50-a294-6d30003bc759","Type":"ContainerDied","Data":"085ee8bdca92e9274ce5d90780a1a341b5e1ff30fbb0fc1eb4f052326e039065"}
Jul 10 18:58:47.415126 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:47.415093    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:47.415282 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:47.415227    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:47.415282 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:47.415272    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:47.415368 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:47.415354    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:48.413827 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:48.413787    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:48.414241 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:48.413897    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:48.682416 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:48.682386    2567 generic.go:356] "Generic (PLEG): container finished" podID="354e5926-7524-4d50-a294-6d30003bc759" containerID="7c933dd08317c90140267c16448a9e1f940d204c7690f91d709c0528f689544a" exitCode=0
Jul 10 18:58:48.682575 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:48.682448    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m2j6" event={"ID":"354e5926-7524-4d50-a294-6d30003bc759","Type":"ContainerDied","Data":"7c933dd08317c90140267c16448a9e1f940d204c7690f91d709c0528f689544a"}
Jul 10 18:58:49.414145 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:49.414115    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:49.414585 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:49.414234    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:49.414585 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:49.414277    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:49.414585 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:49.414396    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:49.932775 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:49.932744    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:49.932964 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:49.932919    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:49.933023 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:49.932986    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs podName:c553a64a-aa13-4a33-90c3-aee6b4f75207 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:21.932967204 +0000 UTC m=+65.184614535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs") pod "network-metrics-daemon-cbt5f" (UID: "c553a64a-aa13-4a33-90c3-aee6b4f75207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:49.981446 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:49.981414    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:50.033432 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:50.033401    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:50.033579 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:50.033451    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr6r\" (UniqueName: \"kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r\") pod \"network-check-target-mgkkh\" (UID: \"76040466-2342-4008-becd-8b68f1f99e5e\") " pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:50.033579 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:50.033544    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:50.033579 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:50.033571    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:50.033734 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:50.033588    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:50.033734 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:50.033600    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret podName:d68b9eae-5b48-4dfa-bfd7-185a928fabd7 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:22.033581847 +0000 UTC m=+65.285229154 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret") pod "global-pull-secret-syncer-4xprr" (UID: "d68b9eae-5b48-4dfa-bfd7-185a928fabd7") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:50.033734 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:50.033601    2567 projected.go:196] Error preparing data for projected volume kube-api-access-7qr6r for pod openshift-network-diagnostics/network-check-target-mgkkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:50.033734 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:50.033633    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r podName:76040466-2342-4008-becd-8b68f1f99e5e nodeName:}" failed. No retries permitted until 2026-07-10 18:59:22.033625917 +0000 UTC m=+65.285273224 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qr6r" (UniqueName: "kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r") pod "network-check-target-mgkkh" (UID: "76040466-2342-4008-becd-8b68f1f99e5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:50.065561 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:50.065536    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:50.333166 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:50.333099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:50.395611 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:50.395578    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:50.413616 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:50.413587    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:50.413716 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:50.413696    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:50.759364 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:50.759268    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:50.860646 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:50.860612    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:50.977369 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:50.977333    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:51.082641 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:51.082609    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:51.175037 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:51.174999    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:51.413718 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:51.413632    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:51.413718 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:51.413632    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:51.413924 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:51.413766    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgkkh" podUID="76040466-2342-4008-becd-8b68f1f99e5e"
Jul 10 18:58:51.413987 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:51.413936    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbt5f" podUID="c553a64a-aa13-4a33-90c3-aee6b4f75207"
Jul 10 18:58:52.413844 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.413812    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:52.414266 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:52.413942    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xprr" podUID="d68b9eae-5b48-4dfa-bfd7-185a928fabd7"
Jul 10 18:58:52.605715 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.605686    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-15.ec2.internal" event="NodeReady"
Jul 10 18:58:52.605859 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.605815    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 10 18:58:52.657217 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.657191    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8446f8d676-trjrq"]
Jul 10 18:58:52.700054 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.700027    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8446f8d676-trjrq"]
Jul 10 18:58:52.700220 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.700066    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tksk4"]
Jul 10 18:58:52.700565 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.700537    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.704635 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.704614    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 10 18:58:52.704767 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.704697    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zn5x2\""
Jul 10 18:58:52.704862 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.704614    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 10 18:58:52.705070 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.705053    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 10 18:58:52.710275 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.710254    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 10 18:58:52.721954 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.721936    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-qpbpd"]
Jul 10 18:58:52.722034 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.721985    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.724765 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.724724    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rqm8x\""
Jul 10 18:58:52.724976 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.724962    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 10 18:58:52.725124 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.725098    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 10 18:58:52.725218 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.725171    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 10 18:58:52.725555 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.725542    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 10 18:58:52.731472 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.731455    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:52.739827 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.739810    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hrppr"]
Jul 10 18:58:52.739943 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.739930    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:52.742727 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.742708    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 10 18:58:52.742923 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.742891    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 10 18:58:52.743061 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.743044    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 10 18:58:52.743163 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.743151    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4dwtj\""
Jul 10 18:58:52.743807 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.743789    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 10 18:58:52.763117 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.763102    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-qpbpd"]
Jul 10 18:58:52.763201 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.763127    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bx94k"]
Jul 10 18:58:52.763251 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.763233    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hrppr"
Jul 10 18:58:52.766222 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.766205    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mwmqx\""
Jul 10 18:58:52.766466 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.766450    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 10 18:58:52.766702 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.766686    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 10 18:58:52.766777 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.766742    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 10 18:58:52.782299 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.782283    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tksk4"]
Jul 10 18:58:52.782371 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.782312    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hrppr"]
Jul 10 18:58:52.782371 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.782326    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bx94k"]
Jul 10 18:58:52.782433 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.782378    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:52.784814 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.784794    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 10 18:58:52.784929 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.784794    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 10 18:58:52.784980 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.784956    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-njnjs\""
Jul 10 18:58:52.817899 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.817877    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:52.855661 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855633    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-installation-pull-secrets\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.855661 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855658    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-bound-sa-token\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.855802 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855680    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw95n\" (UniqueName: \"kubernetes.io/projected/ecff79fd-0d2f-4d7d-8335-63b2b884c39a-kube-api-access-gw95n\") pod \"ingress-canary-hrppr\" (UID: \"ecff79fd-0d2f-4d7d-8335-63b2b884c39a\") " pod="openshift-ingress-canary/ingress-canary-hrppr"
Jul 10 18:58:52.855802 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855694    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/689d980b-bb4b-4e18-86a1-eaea6f78ba18-data-volume\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.855802 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855730    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-image-registry-private-configuration\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.855802 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855769    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.855971 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855800    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecff79fd-0d2f-4d7d-8335-63b2b884c39a-cert\") pod \"ingress-canary-hrppr\" (UID: \"ecff79fd-0d2f-4d7d-8335-63b2b884c39a\") " pod="openshift-ingress-canary/ingress-canary-hrppr"
Jul 10 18:58:52.855971 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855831    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8dc518f-6220-4edc-bb98-4af59a9a65d5-ca-trust-extracted\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.855971 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855888    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4309810b-e277-4090-bb8c-c90f8d7271ee-signing-cabundle\") pod \"service-ca-79f68b9f4f-qpbpd\" (UID: \"4309810b-e277-4090-bb8c-c90f8d7271ee\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:52.855971 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855945    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/689d980b-bb4b-4e18-86a1-eaea6f78ba18-crio-socket\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.855971 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855963    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.856143 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.855986    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.856143 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.856007    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlwnx\" (UniqueName: \"kubernetes.io/projected/4309810b-e277-4090-bb8c-c90f8d7271ee-kube-api-access-nlwnx\") pod \"service-ca-79f68b9f4f-qpbpd\" (UID: \"4309810b-e277-4090-bb8c-c90f8d7271ee\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:52.856143 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.856021    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-certificates\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.856143 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.856037    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4309810b-e277-4090-bb8c-c90f8d7271ee-signing-key\") pod \"service-ca-79f68b9f4f-qpbpd\" (UID: \"4309810b-e277-4090-bb8c-c90f8d7271ee\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:52.856143 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.856057    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9572\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-kube-api-access-w9572\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.856143 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.856101    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8zgb\" (UniqueName: \"kubernetes.io/projected/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-api-access-w8zgb\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.856143 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.856119    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-trusted-ca\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.885990 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.885970    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4xtcp_ebfcae4d-1a86-4c05-b134-59ad0ec54c15/dns-node-resolver/0.log"
Jul 10 18:58:52.894165 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.894141    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:52.957026 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.957100 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957029    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7646e080-f86f-411c-93b0-b57ccfb25013-config-volume\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:52.957100 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecff79fd-0d2f-4d7d-8335-63b2b884c39a-cert\") pod \"ingress-canary-hrppr\" (UID: \"ecff79fd-0d2f-4d7d-8335-63b2b884c39a\") " pod="openshift-ingress-canary/ingress-canary-hrppr"
Jul 10 18:58:52.957100 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957065    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8dc518f-6220-4edc-bb98-4af59a9a65d5-ca-trust-extracted\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.957100 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957083    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4309810b-e277-4090-bb8c-c90f8d7271ee-signing-cabundle\") pod \"service-ca-79f68b9f4f-qpbpd\" (UID: \"4309810b-e277-4090-bb8c-c90f8d7271ee\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:52.957244 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7646e080-f86f-411c-93b0-b57ccfb25013-tmp-dir\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:52.957244 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:52.957131    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:58:52.957244 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957140    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:52.957244 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:52.957146    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8446f8d676-trjrq: secret "image-registry-tls" not found
Jul 10 18:58:52.957244 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957188    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/689d980b-bb4b-4e18-86a1-eaea6f78ba18-crio-socket\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.957244 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:52.957201    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls podName:a8dc518f-6220-4edc-bb98-4af59a9a65d5 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:53.457189078 +0000 UTC m=+36.708836385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls") pod "image-registry-8446f8d676-trjrq" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5") : secret "image-registry-tls" not found
Jul 10 18:58:52.957244 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957228    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.957468 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957254    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.957468 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwnx\" (UniqueName: \"kubernetes.io/projected/4309810b-e277-4090-bb8c-c90f8d7271ee-kube-api-access-nlwnx\") pod \"service-ca-79f68b9f4f-qpbpd\" (UID: \"4309810b-e277-4090-bb8c-c90f8d7271ee\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:52.957468 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957310    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-certificates\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.957468 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957337    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4309810b-e277-4090-bb8c-c90f8d7271ee-signing-key\") pod \"service-ca-79f68b9f4f-qpbpd\" (UID: \"4309810b-e277-4090-bb8c-c90f8d7271ee\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:52.957468 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957361    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9572\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-kube-api-access-w9572\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.957468 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zgb\" (UniqueName: \"kubernetes.io/projected/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-api-access-w8zgb\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.957468 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957407    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8dc518f-6220-4edc-bb98-4af59a9a65d5-ca-trust-extracted\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.957468 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957423    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-trusted-ca\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.957468 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957449    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-installation-pull-secrets\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.957783 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957472    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-bound-sa-token\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.957783 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957504    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw95n\" (UniqueName: \"kubernetes.io/projected/ecff79fd-0d2f-4d7d-8335-63b2b884c39a-kube-api-access-gw95n\") pod \"ingress-canary-hrppr\" (UID: \"ecff79fd-0d2f-4d7d-8335-63b2b884c39a\") " pod="openshift-ingress-canary/ingress-canary-hrppr"
Jul 10 18:58:52.957783 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.957504    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/689d980b-bb4b-4e18-86a1-eaea6f78ba18-crio-socket\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.957980 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:52.957289    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:58:52.957980 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:52.957853    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm podName:689d980b-bb4b-4e18-86a1-eaea6f78ba18 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:53.457837462 +0000 UTC m=+36.709484768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tksk4" (UID: "689d980b-bb4b-4e18-86a1-eaea6f78ba18") : configmap "kube-rbac-proxy" not found
Jul 10 18:58:52.957980 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:52.957352    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:58:52.957980 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:52.957904    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls podName:689d980b-bb4b-4e18-86a1-eaea6f78ba18 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:53.457893427 +0000 UTC m=+36.709540736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tksk4" (UID: "689d980b-bb4b-4e18-86a1-eaea6f78ba18") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:58:52.958189 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.958076    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4309810b-e277-4090-bb8c-c90f8d7271ee-signing-cabundle\") pod \"service-ca-79f68b9f4f-qpbpd\" (UID: \"4309810b-e277-4090-bb8c-c90f8d7271ee\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:52.958189 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.958081    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/689d980b-bb4b-4e18-86a1-eaea6f78ba18-data-volume\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.958189 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.958144    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-image-registry-private-configuration\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.958327 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.958190    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2jk\" (UniqueName: \"kubernetes.io/projected/7646e080-f86f-411c-93b0-b57ccfb25013-kube-api-access-8g2jk\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:52.958421 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.958398    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-certificates\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.958521 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.958493    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/689d980b-bb4b-4e18-86a1-eaea6f78ba18-data-volume\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:52.958590 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.958566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-trusted-ca\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.961541 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.961438    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-image-registry-private-configuration\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.961541 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.961534    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecff79fd-0d2f-4d7d-8335-63b2b884c39a-cert\") pod \"ingress-canary-hrppr\" (UID: \"ecff79fd-0d2f-4d7d-8335-63b2b884c39a\") " pod="openshift-ingress-canary/ingress-canary-hrppr"
Jul 10 18:58:52.961653 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.961468    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-installation-pull-secrets\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.961653 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.961498    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4309810b-e277-4090-bb8c-c90f8d7271ee-signing-key\") pod \"service-ca-79f68b9f4f-qpbpd\" (UID: \"4309810b-e277-4090-bb8c-c90f8d7271ee\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:52.968899 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.968879    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw95n\" (UniqueName: \"kubernetes.io/projected/ecff79fd-0d2f-4d7d-8335-63b2b884c39a-kube-api-access-gw95n\") pod \"ingress-canary-hrppr\" (UID: \"ecff79fd-0d2f-4d7d-8335-63b2b884c39a\") " pod="openshift-ingress-canary/ingress-canary-hrppr"
Jul 10 18:58:52.969457 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.969436    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlwnx\" (UniqueName: \"kubernetes.io/projected/4309810b-e277-4090-bb8c-c90f8d7271ee-kube-api-access-nlwnx\") pod \"service-ca-79f68b9f4f-qpbpd\" (UID: \"4309810b-e277-4090-bb8c-c90f8d7271ee\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:52.969457 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.969448    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9572\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-kube-api-access-w9572\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.969566 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.969552    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-bound-sa-token\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:52.970248 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:52.970232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zgb\" (UniqueName: \"kubernetes.io/projected/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-api-access-w8zgb\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:53.048186 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.048159    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd"
Jul 10 18:58:53.059436 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.059413    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2jk\" (UniqueName: \"kubernetes.io/projected/7646e080-f86f-411c-93b0-b57ccfb25013-kube-api-access-8g2jk\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:53.059523 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.059463    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7646e080-f86f-411c-93b0-b57ccfb25013-config-volume\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:53.059523 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.059485    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7646e080-f86f-411c-93b0-b57ccfb25013-tmp-dir\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:53.059632 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.059608    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:53.059746 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.059729    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:58:53.059799 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.059757    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7646e080-f86f-411c-93b0-b57ccfb25013-tmp-dir\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:53.059799 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.059794    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls podName:7646e080-f86f-411c-93b0-b57ccfb25013 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:53.559777597 +0000 UTC m=+36.811424924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls") pod "dns-default-bx94k" (UID: "7646e080-f86f-411c-93b0-b57ccfb25013") : secret "dns-default-metrics-tls" not found
Jul 10 18:58:53.059923 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.059887    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7646e080-f86f-411c-93b0-b57ccfb25013-config-volume\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:53.067714 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.067698    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2jk\" (UniqueName: \"kubernetes.io/projected/7646e080-f86f-411c-93b0-b57ccfb25013-kube-api-access-8g2jk\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:53.070691 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.070667    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hrppr"
Jul 10 18:58:53.325994 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.325966    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hrppr"]
Jul 10 18:58:53.327865 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.327843    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-qpbpd"]
Jul 10 18:58:53.331214 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:53.331177    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecff79fd_0d2f_4d7d_8335_63b2b884c39a.slice/crio-5a941dbfc9162db0e712a3566d09acb418d8621f85ef95c86b66608558f53903 WatchSource:0}: Error finding container 5a941dbfc9162db0e712a3566d09acb418d8621f85ef95c86b66608558f53903: Status 404 returned error can't find the container with id 5a941dbfc9162db0e712a3566d09acb418d8621f85ef95c86b66608558f53903
Jul 10 18:58:53.331761 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:58:53.331738    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4309810b_e277_4090_bb8c_c90f8d7271ee.slice/crio-0d4f4fdfdd47f3d4082dd23c319958677be2df3b7713138be5c8df354a17f5cc WatchSource:0}: Error finding container 0d4f4fdfdd47f3d4082dd23c319958677be2df3b7713138be5c8df354a17f5cc: Status 404 returned error can't find the container with id 0d4f4fdfdd47f3d4082dd23c319958677be2df3b7713138be5c8df354a17f5cc
Jul 10 18:58:53.416329 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.416311    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:58:53.417004 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.416461    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:58:53.419624 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.419467    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5d8hj\""
Jul 10 18:58:53.419928 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.419889    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:58:53.420331 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.420316    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tr5kt\""
Jul 10 18:58:53.420389 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.420316    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:58:53.420556 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.420543    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:58:53.422007 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.421986    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:53.463558 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.463538    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:53.463636 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.463563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:53.463636 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.463613    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:53.463737 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.463655    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:58:53.463737 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.463690    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:58:53.463737 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.463699    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8446f8d676-trjrq: secret "image-registry-tls" not found
Jul 10 18:58:53.463737 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.463716    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm podName:689d980b-bb4b-4e18-86a1-eaea6f78ba18 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:54.463699369 +0000 UTC m=+37.715346676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tksk4" (UID: "689d980b-bb4b-4e18-86a1-eaea6f78ba18") : configmap "kube-rbac-proxy" not found
Jul 10 18:58:53.463737 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.463733    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls podName:a8dc518f-6220-4edc-bb98-4af59a9a65d5 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:54.463722994 +0000 UTC m=+37.715370301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls") pod "image-registry-8446f8d676-trjrq" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5") : secret "image-registry-tls" not found
Jul 10 18:58:53.463945 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.463757    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:58:53.463945 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.463798    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls podName:689d980b-bb4b-4e18-86a1-eaea6f78ba18 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:54.463786185 +0000 UTC m=+37.715433497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tksk4" (UID: "689d980b-bb4b-4e18-86a1-eaea6f78ba18") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:58:53.523136 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.523110    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:53.564392 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.564366    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:53.564490 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.564474    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:58:53.564546 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:53.564540    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls podName:7646e080-f86f-411c-93b0-b57ccfb25013 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:54.564523011 +0000 UTC m=+37.816170337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls") pod "dns-default-bx94k" (UID: "7646e080-f86f-411c-93b0-b57ccfb25013") : secret "dns-default-metrics-tls" not found
Jul 10 18:58:53.688979 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.688887    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:53.694506 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.694487    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd" event={"ID":"4309810b-e277-4090-bb8c-c90f8d7271ee","Type":"ContainerStarted","Data":"0d4f4fdfdd47f3d4082dd23c319958677be2df3b7713138be5c8df354a17f5cc"}
Jul 10 18:58:53.697417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.697395    2567 generic.go:356] "Generic (PLEG): container finished" podID="354e5926-7524-4d50-a294-6d30003bc759" containerID="dc9881aa29c25fdd479b20aca4c92abe83196263e48055d751d5f516e99c1370" exitCode=0
Jul 10 18:58:53.697491 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.697464    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m2j6" event={"ID":"354e5926-7524-4d50-a294-6d30003bc759","Type":"ContainerDied","Data":"dc9881aa29c25fdd479b20aca4c92abe83196263e48055d751d5f516e99c1370"}
Jul 10 18:58:53.698701 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:53.698677    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hrppr" event={"ID":"ecff79fd-0d2f-4d7d-8335-63b2b884c39a","Type":"ContainerStarted","Data":"5a941dbfc9162db0e712a3566d09acb418d8621f85ef95c86b66608558f53903"}
Jul 10 18:58:54.284821 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:54.284794    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m9r7c_41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54/node-ca/0.log"
Jul 10 18:58:54.413207 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:54.413174    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:58:54.415941 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:54.415901    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:58:54.471729 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:54.471701    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:54.472132 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:54.471743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:54.472132 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:54.471824    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:58:54.472132 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:54.471843    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:58:54.472132 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:54.471894    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm podName:689d980b-bb4b-4e18-86a1-eaea6f78ba18 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:56.471871474 +0000 UTC m=+39.723518794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tksk4" (UID: "689d980b-bb4b-4e18-86a1-eaea6f78ba18") : configmap "kube-rbac-proxy" not found
Jul 10 18:58:54.472132 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:54.471946    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls podName:689d980b-bb4b-4e18-86a1-eaea6f78ba18 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:56.471927969 +0000 UTC m=+39.723575277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tksk4" (UID: "689d980b-bb4b-4e18-86a1-eaea6f78ba18") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:58:54.472132 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:54.471990    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:54.472132 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:54.472088    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:58:54.472132 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:54.472107    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8446f8d676-trjrq: secret "image-registry-tls" not found
Jul 10 18:58:54.472132 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:54.472135    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls podName:a8dc518f-6220-4edc-bb98-4af59a9a65d5 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:56.472126393 +0000 UTC m=+39.723773705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls") pod "image-registry-8446f8d676-trjrq" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5") : secret "image-registry-tls" not found
Jul 10 18:58:54.573309 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:54.573278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:54.573461 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:54.573437    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:58:54.573523 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:54.573512    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls podName:7646e080-f86f-411c-93b0-b57ccfb25013 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:56.573492142 +0000 UTC m=+39.825139452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls") pod "dns-default-bx94k" (UID: "7646e080-f86f-411c-93b0-b57ccfb25013") : secret "dns-default-metrics-tls" not found
Jul 10 18:58:54.705622 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:54.705572    2567 generic.go:356] "Generic (PLEG): container finished" podID="354e5926-7524-4d50-a294-6d30003bc759" containerID="27ecf54c5d14f7ec38b43e0e9b538b03c2049d7c2302174f309bf83319430830" exitCode=0
Jul 10 18:58:54.705622 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:54.705620    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m2j6" event={"ID":"354e5926-7524-4d50-a294-6d30003bc759","Type":"ContainerDied","Data":"27ecf54c5d14f7ec38b43e0e9b538b03c2049d7c2302174f309bf83319430830"}
Jul 10 18:58:55.711735 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:55.711702    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m2j6" event={"ID":"354e5926-7524-4d50-a294-6d30003bc759","Type":"ContainerStarted","Data":"28f404e0685fce02e42da8787ccb1233ce1f5d8686f12524d4770243900f3920"}
Jul 10 18:58:55.736955 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:55.736893    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8m2j6" podStartSLOduration=7.910378506 podStartE2EDuration="38.7368789s" totalImagesPullingTime="30.826500394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:55.735312723 +0000 UTC m=+38.986960053" watchObservedRunningTime="2026-07-10 18:58:55.7368789 +0000 UTC m=+38.988526229"
Jul 10 18:58:55.971247 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:55.971134    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:56.490183 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:56.490151    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:58:56.490346 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:56.490210    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:56.490346 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:56.490232    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:56.490346 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:56.490288    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:58:56.490346 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:56.490306    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8446f8d676-trjrq: secret "image-registry-tls" not found
Jul 10 18:58:56.490346 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:56.490329    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:58:56.490603 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:56.490359    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls podName:a8dc518f-6220-4edc-bb98-4af59a9a65d5 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:00.490343884 +0000 UTC m=+43.741991191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls") pod "image-registry-8446f8d676-trjrq" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5") : secret "image-registry-tls" not found
Jul 10 18:58:56.490603 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:56.490372    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls podName:689d980b-bb4b-4e18-86a1-eaea6f78ba18 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:00.490366142 +0000 UTC m=+43.742013448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tksk4" (UID: "689d980b-bb4b-4e18-86a1-eaea6f78ba18") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:58:56.490820 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:56.490797    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/689d980b-bb4b-4e18-86a1-eaea6f78ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:58:56.590557 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:56.590533    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:58:56.590675 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:56.590658    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:58:56.590720 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:58:56.590711    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls podName:7646e080-f86f-411c-93b0-b57ccfb25013 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:00.590698421 +0000 UTC m=+43.842345728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls") pod "dns-default-bx94k" (UID: "7646e080-f86f-411c-93b0-b57ccfb25013") : secret "dns-default-metrics-tls" not found
Jul 10 18:58:56.870330 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:56.870293    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:56.971863 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:56.971829    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:57.200639 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:57.200587    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:57.716761 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:57.716733    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd" event={"ID":"4309810b-e277-4090-bb8c-c90f8d7271ee","Type":"ContainerStarted","Data":"a50bb6d5f2947cb77222721f91735581e0cc685fd30451d62d12b38c1309e3bf"}
Jul 10 18:58:57.718107 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:57.718083    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hrppr" event={"ID":"ecff79fd-0d2f-4d7d-8335-63b2b884c39a","Type":"ContainerStarted","Data":"1a148d272c20501766694032d5c663e3857eb9e18c7e064263550959e6570cc4"}
Jul 10 18:58:57.737463 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:57.737417    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-qpbpd" podStartSLOduration=2.35825444 podStartE2EDuration="5.737398003s" totalImagesPullingTime="3.379143563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:57.736724911 +0000 UTC m=+40.988372242" watchObservedRunningTime="2026-07-10 18:58:57.737398003 +0000 UTC m=+40.989045333"
Jul 10 18:58:57.755575 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:57.755534    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hrppr" podStartSLOduration=2.381245737 podStartE2EDuration="5.755525121s" totalImagesPullingTime="3.374279384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:57.75457483 +0000 UTC m=+41.006222171" watchObservedRunningTime="2026-07-10 18:58:57.755525121 +0000 UTC m=+41.007172428"
Jul 10 18:58:57.788777 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:57.788745    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:57.884691 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:57.884646    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:57.983924 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:57.983821    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:58.089694 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:58.089663    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:58.187896 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:58.187866    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:58.282241 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:58.282144    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:59.010297 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:59.010259    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:59.416057 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:59.416031    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:59.802005 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:58:59.801967    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:00.202178 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:00.202106    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:00.302613 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:00.302583    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:00.402763 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:00.402725    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:00.519572 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:00.519506    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:59:00.519572 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:00.519567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:59:00.519728 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:59:00.519641    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:00.519728 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:59:00.519659    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:59:00.519728 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:59:00.519675    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8446f8d676-trjrq: secret "image-registry-tls" not found
Jul 10 18:59:00.519728 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:59:00.519700    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls podName:689d980b-bb4b-4e18-86a1-eaea6f78ba18 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:08.519685317 +0000 UTC m=+51.771332623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tksk4" (UID: "689d980b-bb4b-4e18-86a1-eaea6f78ba18") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:00.519728 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:59:00.519714    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls podName:a8dc518f-6220-4edc-bb98-4af59a9a65d5 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:08.51970816 +0000 UTC m=+51.771355466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls") pod "image-registry-8446f8d676-trjrq" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5") : secret "image-registry-tls" not found
Jul 10 18:59:00.601012 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:00.600989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:00.620219 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:00.620191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:59:00.620344 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:59:00.620329    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:00.620403 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:59:00.620382    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls podName:7646e080-f86f-411c-93b0-b57ccfb25013 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:08.620369022 +0000 UTC m=+51.872016330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls") pod "dns-default-bx94k" (UID: "7646e080-f86f-411c-93b0-b57ccfb25013") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:01.002356 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:01.002326    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:01.204674 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:01.204635    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:01.404413 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:01.404386    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:01.806070 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:01.806041    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:01.868256 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:01.868232    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:02.021137 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:02.021101    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:02.602684 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:02.602652    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:02.666288 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:02.666258    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:03.006391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:03.006289    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:03.198389 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:03.198357    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:03.299653 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:03.299623    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:03.361023 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:03.360999    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:03.425170 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:03.425136    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:04.600527 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:04.600498    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:04.815260 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:04.815229    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:05.008059 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:05.007984    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:05.069012 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:05.068989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:05.199174 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:05.199145    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:05.401469 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:05.401435    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:05.562690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:05.562658    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:06.201180 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:06.201150    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:06.605012 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:06.604973    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:07.002150 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:07.002082    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:07.405340 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:07.405312    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:07.610557 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:07.610526    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:08.196038 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.196008    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:08.406505 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.406480    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:08.586250 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.586222    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:59:08.586375 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.586277    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:59:08.589657 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.589632    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls\") pod \"image-registry-8446f8d676-trjrq\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") " pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:59:08.589758 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.589671    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/689d980b-bb4b-4e18-86a1-eaea6f78ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tksk4\" (UID: \"689d980b-bb4b-4e18-86a1-eaea6f78ba18\") " pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:59:08.610478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.610457    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:59:08.628336 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.628309    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:08.630196 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.630182    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tksk4"
Jul 10 18:59:08.690752 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.690291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:59:08.695344 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.695277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7646e080-f86f-411c-93b0-b57ccfb25013-metrics-tls\") pod \"dns-default-bx94k\" (UID: \"7646e080-f86f-411c-93b0-b57ccfb25013\") " pod="openshift-dns/dns-default-bx94k"
Jul 10 18:59:08.787171 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.787135    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8446f8d676-trjrq"]
Jul 10 18:59:08.790223 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:08.790198    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dc518f_6220_4edc_bb98_4af59a9a65d5.slice/crio-85e25348437f78d04bc40397209af2f5cced064e1c6e54584619934a94cdb6ef WatchSource:0}: Error finding container 85e25348437f78d04bc40397209af2f5cced064e1c6e54584619934a94cdb6ef: Status 404 returned error can't find the container with id 85e25348437f78d04bc40397209af2f5cced064e1c6e54584619934a94cdb6ef
Jul 10 18:59:08.806106 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.806082    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tksk4"]
Jul 10 18:59:08.808492 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:08.808470    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689d980b_bb4b_4e18_86a1_eaea6f78ba18.slice/crio-f9dded5867cd0dba171553ff8779c25379da4405d5ae586330957a17c4b63f5e WatchSource:0}: Error finding container f9dded5867cd0dba171553ff8779c25379da4405d5ae586330957a17c4b63f5e: Status 404 returned error can't find the container with id f9dded5867cd0dba171553ff8779c25379da4405d5ae586330957a17c4b63f5e
Jul 10 18:59:08.992821 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:08.992797    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bx94k"
Jul 10 18:59:09.138828 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:09.138802    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bx94k"]
Jul 10 18:59:09.141302 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:09.141271    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7646e080_f86f_411c_93b0_b57ccfb25013.slice/crio-573b52ad92c6f59039067e3eafc76129e5ce87b2ea355b7523ee94c80c7d4c81 WatchSource:0}: Error finding container 573b52ad92c6f59039067e3eafc76129e5ce87b2ea355b7523ee94c80c7d4c81: Status 404 returned error can't find the container with id 573b52ad92c6f59039067e3eafc76129e5ce87b2ea355b7523ee94c80c7d4c81
Jul 10 18:59:09.751555 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:09.751515    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tksk4" event={"ID":"689d980b-bb4b-4e18-86a1-eaea6f78ba18","Type":"ContainerStarted","Data":"f9dded5867cd0dba171553ff8779c25379da4405d5ae586330957a17c4b63f5e"}
Jul 10 18:59:09.753309 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:09.753279    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bx94k" event={"ID":"7646e080-f86f-411c-93b0-b57ccfb25013","Type":"ContainerStarted","Data":"573b52ad92c6f59039067e3eafc76129e5ce87b2ea355b7523ee94c80c7d4c81"}
Jul 10 18:59:09.755777 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:09.755706    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8446f8d676-trjrq" event={"ID":"a8dc518f-6220-4edc-bb98-4af59a9a65d5","Type":"ContainerStarted","Data":"a844c9f63544e1703f206c9fc4e8f04b9b0fcfa04dd644878275c0bf0a0906d7"}
Jul 10 18:59:09.755777 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:09.755750    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8446f8d676-trjrq" event={"ID":"a8dc518f-6220-4edc-bb98-4af59a9a65d5","Type":"ContainerStarted","Data":"85e25348437f78d04bc40397209af2f5cced064e1c6e54584619934a94cdb6ef"}
Jul 10 18:59:09.756280 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:09.756031    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:59:09.789988 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:09.789039    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8446f8d676-trjrq" podStartSLOduration=53.789020485 podStartE2EDuration="53.789020485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:09.787223027 +0000 UTC m=+53.038870355" watchObservedRunningTime="2026-07-10 18:59:09.789020485 +0000 UTC m=+53.040667817"
Jul 10 18:59:13.769963 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:13.769933    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tksk4" event={"ID":"689d980b-bb4b-4e18-86a1-eaea6f78ba18","Type":"ContainerStarted","Data":"af7af66f54bab0ae64b9993c2471e08579570a0860e53e82914f262ec4254eb7"}
Jul 10 18:59:13.771607 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:13.771587    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bx94k" event={"ID":"7646e080-f86f-411c-93b0-b57ccfb25013","Type":"ContainerStarted","Data":"5ccbe5421391874324d3462a38df741d67ba7f5b400878a1c68e794176bef852"}
Jul 10 18:59:13.771689 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:13.771626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bx94k" event={"ID":"7646e080-f86f-411c-93b0-b57ccfb25013","Type":"ContainerStarted","Data":"72348f31f81be08f96f5cc711d5c01bad227d54284deba0755ab6af24fc09469"}
Jul 10 18:59:13.771749 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:13.771730    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bx94k"
Jul 10 18:59:13.791391 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:13.791327    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bx94k" podStartSLOduration=17.932069739 podStartE2EDuration="21.791293708s" totalImagesPullingTime="3.859223969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:13.790086659 +0000 UTC m=+57.041733999" watchObservedRunningTime="2026-07-10 18:59:13.791293708 +0000 UTC m=+57.042941015"
Jul 10 18:59:14.778642 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:14.778604    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tksk4" event={"ID":"689d980b-bb4b-4e18-86a1-eaea6f78ba18","Type":"ContainerStarted","Data":"6c5f568931a67471410f431263bf2b929241a02de963e3b817f1f3b1f96bd534"}
Jul 10 18:59:14.996362 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:14.996328    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8446f8d676-trjrq"]
Jul 10 18:59:15.110577 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:15.110546    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:15.225313 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:15.225285    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:15.415666 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:15.415580    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:15.515287 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:15.515258    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:17.141744 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:17.141676    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:17.262793 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:17.262766    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:17.341729 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:17.341701    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:17.689132 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:17.689109    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdfcs"
Jul 10 18:59:17.790579 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:17.790553    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tksk4" event={"ID":"689d980b-bb4b-4e18-86a1-eaea6f78ba18","Type":"ContainerStarted","Data":"9f4ead316db593f91a38673a13ee29720d2107f952079b72ffd8e094663ed95c"}
Jul 10 18:59:17.815442 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:17.815399    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tksk4" podStartSLOduration=17.909943884 podStartE2EDuration="25.815385221s" totalImagesPullingTime="7.905441337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:17.815091638 +0000 UTC m=+61.066738969" watchObservedRunningTime="2026-07-10 18:59:17.815385221 +0000 UTC m=+61.067032549"
Jul 10 18:59:17.891971 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:17.891936    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:17.992173 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:17.992117    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:18.062264 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:18.062239    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:18.772019 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:18.771992    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:18.872885 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:18.872851    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:18.971340 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:18.971308    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:19.193083 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:19.193058    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:19.271019 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:19.270990    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:19.432020 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:19.431976    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:20.392663 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:20.392629    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:20.480616 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:20.480585    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:20.602210 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:20.602184    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:20.968320 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:20.968297    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:21.037269 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:21.037237    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:21.992179 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:21.992145    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:59:21.994857 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:21.994840    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:59:22.005043 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.005017    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c553a64a-aa13-4a33-90c3-aee6b4f75207-metrics-certs\") pod \"network-metrics-daemon-cbt5f\" (UID: \"c553a64a-aa13-4a33-90c3-aee6b4f75207\") " pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:59:22.093060 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.093029    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr6r\" (UniqueName: \"kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r\") pod \"network-check-target-mgkkh\" (UID: \"76040466-2342-4008-becd-8b68f1f99e5e\") " pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:59:22.093212 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.093098    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:59:22.095502 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.095483    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:59:22.095619 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.095584    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:59:22.105357 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.105332    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d68b9eae-5b48-4dfa-bfd7-185a928fabd7-original-pull-secret\") pod \"global-pull-secret-syncer-4xprr\" (UID: \"d68b9eae-5b48-4dfa-bfd7-185a928fabd7\") " pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:59:22.106105 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.106088    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:59:22.115534 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.115513    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qr6r\" (UniqueName: \"kubernetes.io/projected/76040466-2342-4008-becd-8b68f1f99e5e-kube-api-access-7qr6r\") pod \"network-check-target-mgkkh\" (UID: \"76040466-2342-4008-becd-8b68f1f99e5e\") " pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:59:22.246297 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.246251    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tr5kt\""
Jul 10 18:59:22.251325 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.251306    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5d8hj\""
Jul 10 18:59:22.254167 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.254144    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbt5f"
Jul 10 18:59:22.259922 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.259897    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:59:22.324466 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.324441    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xprr"
Jul 10 18:59:22.387096 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.387027    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cbt5f"]
Jul 10 18:59:22.390881 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:22.390853    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc553a64a_aa13_4a33_90c3_aee6b4f75207.slice/crio-8aa4a1d1ea72d14989b48658edc622ca4bc73337cd88213bcec8b248064eb822 WatchSource:0}: Error finding container 8aa4a1d1ea72d14989b48658edc622ca4bc73337cd88213bcec8b248064eb822: Status 404 returned error can't find the container with id 8aa4a1d1ea72d14989b48658edc622ca4bc73337cd88213bcec8b248064eb822
Jul 10 18:59:22.402797 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.402773    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mgkkh"]
Jul 10 18:59:22.405819 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:22.405795    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76040466_2342_4008_becd_8b68f1f99e5e.slice/crio-3312b24480b141d08d2d2375131371c584804fa7b21c18d6cc879f06f8aa60d4 WatchSource:0}: Error finding container 3312b24480b141d08d2d2375131371c584804fa7b21c18d6cc879f06f8aa60d4: Status 404 returned error can't find the container with id 3312b24480b141d08d2d2375131371c584804fa7b21c18d6cc879f06f8aa60d4
Jul 10 18:59:22.447258 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.447227    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4xprr"]
Jul 10 18:59:22.450613 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:22.450590    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68b9eae_5b48_4dfa_bfd7_185a928fabd7.slice/crio-67ee65220fe02bb8a737429b8ca64962858f2833bca98c6f92289c95e7f8c4ec WatchSource:0}: Error finding container 67ee65220fe02bb8a737429b8ca64962858f2833bca98c6f92289c95e7f8c4ec: Status 404 returned error can't find the container with id 67ee65220fe02bb8a737429b8ca64962858f2833bca98c6f92289c95e7f8c4ec
Jul 10 18:59:22.809195 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.809162    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mgkkh" event={"ID":"76040466-2342-4008-becd-8b68f1f99e5e","Type":"ContainerStarted","Data":"3312b24480b141d08d2d2375131371c584804fa7b21c18d6cc879f06f8aa60d4"}
Jul 10 18:59:22.810280 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.810258    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cbt5f" event={"ID":"c553a64a-aa13-4a33-90c3-aee6b4f75207","Type":"ContainerStarted","Data":"8aa4a1d1ea72d14989b48658edc622ca4bc73337cd88213bcec8b248064eb822"}
Jul 10 18:59:22.811332 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:22.811306    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4xprr" event={"ID":"d68b9eae-5b48-4dfa-bfd7-185a928fabd7","Type":"ContainerStarted","Data":"67ee65220fe02bb8a737429b8ca64962858f2833bca98c6f92289c95e7f8c4ec"}
Jul 10 18:59:23.019525 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.017670    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:23.078227 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.078074    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:23.082199 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.081125    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-tm824"]
Jul 10 18:59:23.084425 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.084403    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.087263 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.087001    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 10 18:59:23.087263 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.087217    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 10 18:59:23.087787 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.087508    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9k4kv\""
Jul 10 18:59:23.087787 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.087651    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 10 18:59:23.088061 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.087954    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 10 18:59:23.088158 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.088095    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 10 18:59:23.093798 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.093745    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-tm824"]
Jul 10 18:59:23.186948 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.186898    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:23.202246 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.202214    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6066461-2dd6-4e31-b512-7b1502f23a96-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.202364 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.202277    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxqw\" (UniqueName: \"kubernetes.io/projected/d6066461-2dd6-4e31-b512-7b1502f23a96-kube-api-access-kdxqw\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.202430 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.202378    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6066461-2dd6-4e31-b512-7b1502f23a96-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.202430 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.202412    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6066461-2dd6-4e31-b512-7b1502f23a96-metrics-client-ca\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.303692 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.303644    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6066461-2dd6-4e31-b512-7b1502f23a96-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.303692 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.303694    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6066461-2dd6-4e31-b512-7b1502f23a96-metrics-client-ca\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.303925 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.303754    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6066461-2dd6-4e31-b512-7b1502f23a96-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.303925 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.303779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxqw\" (UniqueName: \"kubernetes.io/projected/d6066461-2dd6-4e31-b512-7b1502f23a96-kube-api-access-kdxqw\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.305127 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.304764    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6066461-2dd6-4e31-b512-7b1502f23a96-metrics-client-ca\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.307617 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.307571    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6066461-2dd6-4e31-b512-7b1502f23a96-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.308571 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.308547    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6066461-2dd6-4e31-b512-7b1502f23a96-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.312225 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.312198    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxqw\" (UniqueName: \"kubernetes.io/projected/d6066461-2dd6-4e31-b512-7b1502f23a96-kube-api-access-kdxqw\") pod \"prometheus-operator-64785b6675-tm824\" (UID: \"d6066461-2dd6-4e31-b512-7b1502f23a96\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.399299 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.399228    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-tm824"
Jul 10 18:59:23.734569 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.734534    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-tm824"]
Jul 10 18:59:23.780671 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:23.780638    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bx94k"
Jul 10 18:59:23.854734 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:23.854698    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6066461_2dd6_4e31_b512_7b1502f23a96.slice/crio-87dea6f64895e9aeaf157cb75cb16965b03dc4d65e05ce49d35587500423f575 WatchSource:0}: Error finding container 87dea6f64895e9aeaf157cb75cb16965b03dc4d65e05ce49d35587500423f575: Status 404 returned error can't find the container with id 87dea6f64895e9aeaf157cb75cb16965b03dc4d65e05ce49d35587500423f575
Jul 10 18:59:24.822500 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:24.822463    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-tm824" event={"ID":"d6066461-2dd6-4e31-b512-7b1502f23a96","Type":"ContainerStarted","Data":"87dea6f64895e9aeaf157cb75cb16965b03dc4d65e05ce49d35587500423f575"}
Jul 10 18:59:24.824943 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:24.824900    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cbt5f" event={"ID":"c553a64a-aa13-4a33-90c3-aee6b4f75207","Type":"ContainerStarted","Data":"362cd6fdaaa6878ce0c0158ae450228fdfba34ff89fcacf00b4e4ffa2c79ea7b"}
Jul 10 18:59:24.825075 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:24.824968    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cbt5f" event={"ID":"c553a64a-aa13-4a33-90c3-aee6b4f75207","Type":"ContainerStarted","Data":"66097a8d738909cb43f11e4475a57867deb38b1e73cd9b7e258eaee53a46b15a"}
Jul 10 18:59:24.843268 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:24.843218    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cbt5f" podStartSLOduration=66.353725657 podStartE2EDuration="1m7.843199032s" totalImagesPullingTime="1.489473375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:24.842623752 +0000 UTC m=+68.094271082" watchObservedRunningTime="2026-07-10 18:59:24.843199032 +0000 UTC m=+68.094846362"
Jul 10 18:59:24.951461 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:24.951413    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:25.003121 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.003083    2567 patch_prober.go:28] interesting pod/image-registry-8446f8d676-trjrq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 10 18:59:25.003297 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.003146    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8446f8d676-trjrq" podUID="a8dc518f-6220-4edc-bb98-4af59a9a65d5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:59:25.053283 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.053185    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:25.868370 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.868338    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-96gct"]
Jul 10 18:59:25.870620 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.870592    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-96gct"
Jul 10 18:59:25.873363 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.873213    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 10 18:59:25.873363 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.873240    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-brrzq\""
Jul 10 18:59:25.873363 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.873260    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 10 18:59:25.884332 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.884309    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-96gct"]
Jul 10 18:59:25.885066 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.885025    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:25.928586 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.928556    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gcm5\" (UniqueName: \"kubernetes.io/projected/116f558a-6e70-474a-83d8-fa124957af41-kube-api-access-5gcm5\") pod \"downloads-68c9cbb497-96gct\" (UID: \"116f558a-6e70-474a-83d8-fa124957af41\") " pod="openshift-console/downloads-68c9cbb497-96gct"
Jul 10 18:59:25.984168 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:25.984137    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:26.029558 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:26.029520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gcm5\" (UniqueName: \"kubernetes.io/projected/116f558a-6e70-474a-83d8-fa124957af41-kube-api-access-5gcm5\") pod \"downloads-68c9cbb497-96gct\" (UID: \"116f558a-6e70-474a-83d8-fa124957af41\") " pod="openshift-console/downloads-68c9cbb497-96gct"
Jul 10 18:59:26.038527 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:26.038498    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gcm5\" (UniqueName: \"kubernetes.io/projected/116f558a-6e70-474a-83d8-fa124957af41-kube-api-access-5gcm5\") pod \"downloads-68c9cbb497-96gct\" (UID: \"116f558a-6e70-474a-83d8-fa124957af41\") " pod="openshift-console/downloads-68c9cbb497-96gct"
Jul 10 18:59:26.044342 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:26.044320    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:26.181497 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:26.181421    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-96gct"
Jul 10 18:59:26.184319 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:26.184215    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:26.269274 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:26.269245    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:26.406675 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:26.406648    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:26.472668 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:26.472576    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:26.728779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:26.728707    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:26.830008 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:26.829897    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:27.187496 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.187466    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:27.262963 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.262932    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:27.523287 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.523268    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-96gct"]
Jul 10 18:59:27.527822 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:27.526558    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116f558a_6e70_474a_83d8_fa124957af41.slice/crio-eaf9033c646569d44afbb2e95bb8f2f49ad0161d93e2d43ca2906cd66e24e44f WatchSource:0}: Error finding container eaf9033c646569d44afbb2e95bb8f2f49ad0161d93e2d43ca2906cd66e24e44f: Status 404 returned error can't find the container with id eaf9033c646569d44afbb2e95bb8f2f49ad0161d93e2d43ca2906cd66e24e44f
Jul 10 18:59:27.545882 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.545851    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:27.753049 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.752967    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:27.836839 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.836809    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-96gct" event={"ID":"116f558a-6e70-474a-83d8-fa124957af41","Type":"ContainerStarted","Data":"eaf9033c646569d44afbb2e95bb8f2f49ad0161d93e2d43ca2906cd66e24e44f"}
Jul 10 18:59:27.838199 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.838172    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4xprr" event={"ID":"d68b9eae-5b48-4dfa-bfd7-185a928fabd7","Type":"ContainerStarted","Data":"36bef1443acfc6b0bf31e6be2e1eb15437e04ed4241417e20faf389e32040908"}
Jul 10 18:59:27.840045 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.840024    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-tm824" event={"ID":"d6066461-2dd6-4e31-b512-7b1502f23a96","Type":"ContainerStarted","Data":"7c50d61ae77eaa8038f524477ba8f969651f7cc09de056f088fd758d2e88c6d3"}
Jul 10 18:59:27.840156 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.840069    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-tm824" event={"ID":"d6066461-2dd6-4e31-b512-7b1502f23a96","Type":"ContainerStarted","Data":"054c46b1dff212ec41041818a048d3da1190cbd1e67962636ba5a3c0a7062e3c"}
Jul 10 18:59:27.841333 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.841314    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mgkkh" event={"ID":"76040466-2342-4008-becd-8b68f1f99e5e","Type":"ContainerStarted","Data":"cb438d310bf40b74725f60216a48f2d48718541e571cd530a65dba7214e40ef7"}
Jul 10 18:59:27.841450 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.841438    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 18:59:27.862129 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.862093    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4xprr" podStartSLOduration=65.913369229 podStartE2EDuration="1m10.862080821s" totalImagesPullingTime="4.948711592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:27.861338783 +0000 UTC m=+71.112986122" watchObservedRunningTime="2026-07-10 18:59:27.862080821 +0000 UTC m=+71.113728150"
Jul 10 18:59:27.881744 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.881722    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:27.891823 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:27.891781    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-tm824" podStartSLOduration=1.384913238 podStartE2EDuration="4.891770049s" totalImagesPullingTime="3.506856811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:27.890941495 +0000 UTC m=+71.142588823" watchObservedRunningTime="2026-07-10 18:59:27.891770049 +0000 UTC m=+71.143417377"
Jul 10 18:59:28.069666 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:28.069640    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:29.258067 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.258034    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:29.346345 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.346309    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:29.447118 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.447035    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:29.531231 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.530954    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mgkkh" podStartSLOduration=67.555335481 podStartE2EDuration="1m12.530933223s" totalImagesPullingTime="4.975597742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:27.932005651 +0000 UTC m=+71.183652984" watchObservedRunningTime="2026-07-10 18:59:29.530933223 +0000 UTC m=+72.782580547"
Jul 10 18:59:29.532411 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.532388    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sxg79"]
Jul 10 18:59:29.534790 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.534769    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.537506 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.537296    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hl7tz\""
Jul 10 18:59:29.537506 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.537346    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 10 18:59:29.537654 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.537524    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 10 18:59:29.538456 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.538436    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 10 18:59:29.555486 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.555448    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:29.650998 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.650970    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:29.659961 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.659935    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61e594fb-2523-455e-b91d-5b9662e902e1-sys\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.660076 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.659972    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-wtmp\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.660076 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.659999    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.660076 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.660040    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61e594fb-2523-455e-b91d-5b9662e902e1-root\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.660228 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.660083    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-tls\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.660228 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.660111    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ncfw\" (UniqueName: \"kubernetes.io/projected/61e594fb-2523-455e-b91d-5b9662e902e1-kube-api-access-7ncfw\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.660228 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.660140    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-textfile\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.660228 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.660176    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e594fb-2523-455e-b91d-5b9662e902e1-metrics-client-ca\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.660228 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.660208    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.760661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ncfw\" (UniqueName: \"kubernetes.io/projected/61e594fb-2523-455e-b91d-5b9662e902e1-kube-api-access-7ncfw\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.760711    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-textfile\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.760743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e594fb-2523-455e-b91d-5b9662e902e1-metrics-client-ca\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.760772    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.760800    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61e594fb-2523-455e-b91d-5b9662e902e1-sys\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.760824    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-wtmp\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.760850    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.760886    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61e594fb-2523-455e-b91d-5b9662e902e1-root\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.760945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-tls\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:59:29.761071    2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 10 18:59:29.761604 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:59:29.761133    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-tls podName:61e594fb-2523-455e-b91d-5b9662e902e1 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:30.26111143 +0000 UTC m=+73.512758739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-tls") pod "node-exporter-sxg79" (UID: "61e594fb-2523-455e-b91d-5b9662e902e1") : secret "node-exporter-tls" not found
Jul 10 18:59:29.762365 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.761798    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-textfile\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.762365 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.761977    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61e594fb-2523-455e-b91d-5b9662e902e1-root\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.762365 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.762101    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-wtmp\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.762365 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.762141    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61e594fb-2523-455e-b91d-5b9662e902e1-sys\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.763081 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.763026    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e594fb-2523-455e-b91d-5b9662e902e1-metrics-client-ca\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.763599 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.763574    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.764366 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.764321    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.774760 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.774731    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ncfw\" (UniqueName: \"kubernetes.io/projected/61e594fb-2523-455e-b91d-5b9662e902e1-kube-api-access-7ncfw\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:29.907131 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:29.903854    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:30.001891 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.001856    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:30.109656 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.109571    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:30.202493 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.202416    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:30.264807 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.264779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-tls\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:30.266896 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.266874    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61e594fb-2523-455e-b91d-5b9662e902e1-node-exporter-tls\") pod \"node-exporter-sxg79\" (UID: \"61e594fb-2523-455e-b91d-5b9662e902e1\") " pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:30.316453 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.316421    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:30.411095 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.411069    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:30.449301 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.449227    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sxg79"
Jul 10 18:59:30.489957 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:30.489932    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e594fb_2523_455e_b91d_5b9662e902e1.slice/crio-bfef4b76f538eed8df5c819f0dfd8a4d46583e818aaa93d690c1bd8701372e55 WatchSource:0}: Error finding container bfef4b76f538eed8df5c819f0dfd8a4d46583e818aaa93d690c1bd8701372e55: Status 404 returned error can't find the container with id bfef4b76f538eed8df5c819f0dfd8a4d46583e818aaa93d690c1bd8701372e55
Jul 10 18:59:30.514551 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.514523    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:30.603825 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.603794    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:30.713638 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.713608    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:30.857611 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:30.857574    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxg79" event={"ID":"61e594fb-2523-455e-b91d-5b9662e902e1","Type":"ContainerStarted","Data":"bfef4b76f538eed8df5c819f0dfd8a4d46583e818aaa93d690c1bd8701372e55"}
Jul 10 18:59:31.537086 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.537051    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77c964dcc5-89zt7"]
Jul 10 18:59:31.539586 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.539563    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.543960 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.543900    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 10 18:59:31.544145 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.544077    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 10 18:59:31.545082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.544931    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 10 18:59:31.545082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.544981    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-f54tq\""
Jul 10 18:59:31.545082 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.544992    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-35v3b91p3auk9\""
Jul 10 18:59:31.545312 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.545183    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 10 18:59:31.545312 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.545212    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 10 18:59:31.559607 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.559584    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77c964dcc5-89zt7"]
Jul 10 18:59:31.675770 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.675740    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.675935 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.675792    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.675935 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.675836    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-tls\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.675935 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.675862    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rk9g\" (UniqueName: \"kubernetes.io/projected/b6c7a7f8-98f7-4469-a203-be4c8eb99869-kube-api-access-6rk9g\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.676095 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.675954    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6c7a7f8-98f7-4469-a203-be4c8eb99869-metrics-client-ca\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.676095 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.675991    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.676095 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.676016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.676095 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.676038    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-grpc-tls\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.777067 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.777029    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6c7a7f8-98f7-4469-a203-be4c8eb99869-metrics-client-ca\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.777182 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.777081    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.777182 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.777114    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.777182 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.777147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-grpc-tls\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.777377 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.777184    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.777424 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.777366    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.777470 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.777420    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-tls\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.777470 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.777450    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rk9g\" (UniqueName: \"kubernetes.io/projected/b6c7a7f8-98f7-4469-a203-be4c8eb99869-kube-api-access-6rk9g\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.777887 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.777836    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6c7a7f8-98f7-4469-a203-be4c8eb99869-metrics-client-ca\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.780188 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.780142    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.780289 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.780225    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.780354 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.780331    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.780640 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.780613    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-tls\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.780888 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.780863    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-grpc-tls\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.781304 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.781285    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6c7a7f8-98f7-4469-a203-be4c8eb99869-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.786854 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.786827    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rk9g\" (UniqueName: \"kubernetes.io/projected/b6c7a7f8-98f7-4469-a203-be4c8eb99869-kube-api-access-6rk9g\") pod \"thanos-querier-77c964dcc5-89zt7\" (UID: \"b6c7a7f8-98f7-4469-a203-be4c8eb99869\") " pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.855344 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.855321    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:31.863992 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.863061    2567 generic.go:356] "Generic (PLEG): container finished" podID="61e594fb-2523-455e-b91d-5b9662e902e1" containerID="3f5a1da7dcba386e3b70ccdbba9311a378ed5cf74443d7cb8a6df7e5e05ba55d" exitCode=0
Jul 10 18:59:31.863992 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:31.863185    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxg79" event={"ID":"61e594fb-2523-455e-b91d-5b9662e902e1","Type":"ContainerDied","Data":"3f5a1da7dcba386e3b70ccdbba9311a378ed5cf74443d7cb8a6df7e5e05ba55d"}
Jul 10 18:59:32.011896 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:32.011869    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77c964dcc5-89zt7"]
Jul 10 18:59:32.014440 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:32.014414    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c7a7f8_98f7_4469_a203_be4c8eb99869.slice/crio-7d53bf199b21c1a39b2c8afceae6b6c02344f71b91a6a52da783f03a331586b4 WatchSource:0}: Error finding container 7d53bf199b21c1a39b2c8afceae6b6c02344f71b91a6a52da783f03a331586b4: Status 404 returned error can't find the container with id 7d53bf199b21c1a39b2c8afceae6b6c02344f71b91a6a52da783f03a331586b4
Jul 10 18:59:32.560985 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:32.560891    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:32.644704 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:32.644677    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:32.832662 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:32.832585    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:32.870028 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:32.869782    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxg79" event={"ID":"61e594fb-2523-455e-b91d-5b9662e902e1","Type":"ContainerStarted","Data":"f5f41109c459cff0c75a5d78af9926b54159f016eee028058f2d9befd38904da"}
Jul 10 18:59:32.870028 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:32.869850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxg79" event={"ID":"61e594fb-2523-455e-b91d-5b9662e902e1","Type":"ContainerStarted","Data":"1580c952e7ae9e5b07bf7a0bdfbcb6b4b9f1010c3b50778906ac597e60d0d02d"}
Jul 10 18:59:32.871685 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:32.871659    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7" event={"ID":"b6c7a7f8-98f7-4469-a203-be4c8eb99869","Type":"ContainerStarted","Data":"7d53bf199b21c1a39b2c8afceae6b6c02344f71b91a6a52da783f03a331586b4"}
Jul 10 18:59:32.891731 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:32.891666    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sxg79" podStartSLOduration=2.76072219 podStartE2EDuration="3.891647612s" totalImagesPullingTime="1.056096284s" totalInitContainerRuntime="74.829138ms" isStatefulPod=true podCreationTimestamp="2026-07-10 18:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:32.889761592 +0000 UTC m=+76.141408923" watchObservedRunningTime="2026-07-10 18:59:32.891647612 +0000 UTC m=+76.143294923"
Jul 10 18:59:32.937779 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:32.937455    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:33.034227 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:33.034193    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:33.170407 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:33.170330    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:33.722681 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:33.722633    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:33.824120 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:33.824091    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:34.262322 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.262295    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:34.360377 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.360302    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:34.675867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.675830    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:34.833319 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.833288    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"]
Jul 10 18:59:34.835884 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.835863    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:34.838957 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.838930    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 10 18:59:34.839073 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.839020    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 10 18:59:34.839136 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.839088    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-p8nrn\""
Jul 10 18:59:34.839531 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.839348    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 10 18:59:34.839531 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.839380    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 10 18:59:34.839823 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.839805    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 10 18:59:34.845824 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.845803    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 10 18:59:34.860994 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.860975    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"]
Jul 10 18:59:34.882162 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.882135    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7" event={"ID":"b6c7a7f8-98f7-4469-a203-be4c8eb99869","Type":"ContainerStarted","Data":"36f71bdf7241d5ec208fdeba6a6a8d0ca81276d73ce03b4a13884ebc872c503c"}
Jul 10 18:59:34.882244 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.882194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7" event={"ID":"b6c7a7f8-98f7-4469-a203-be4c8eb99869","Type":"ContainerStarted","Data":"524cb11d848d82daeafed5d029fc00dd442693314089b5bf417b2fd053e342a5"}
Jul 10 18:59:34.882244 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.882232    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7" event={"ID":"b6c7a7f8-98f7-4469-a203-be4c8eb99869","Type":"ContainerStarted","Data":"1f97fa8dcb3439ae0faebba67b5347068443340e5ce04f9e483ebaecf0498422"}
Jul 10 18:59:34.908150 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.908118    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5356aebe-d543-4fd8-a974-b92ec16ed048-serving-certs-ca-bundle\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:34.908238 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.908176    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5356aebe-d543-4fd8-a974-b92ec16ed048-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:34.908238 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.908203    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-secret-telemeter-client\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:34.908238 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.908229    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-federate-client-tls\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:34.908373 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.908253    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbsvh\" (UniqueName: \"kubernetes.io/projected/5356aebe-d543-4fd8-a974-b92ec16ed048-kube-api-access-rbsvh\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:34.908373 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.908284    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:34.908373 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.908332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-telemeter-client-tls\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:34.908373 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:34.908357    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5356aebe-d543-4fd8-a974-b92ec16ed048-metrics-client-ca\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.002677 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.002604    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:59:35.009200 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.009173    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5356aebe-d543-4fd8-a974-b92ec16ed048-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.009325 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.009213    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-secret-telemeter-client\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.009325 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.009241    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-federate-client-tls\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.009325 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.009267    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbsvh\" (UniqueName: \"kubernetes.io/projected/5356aebe-d543-4fd8-a974-b92ec16ed048-kube-api-access-rbsvh\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.009325 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.009300    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.009731 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.009585    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-telemeter-client-tls\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.009731 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.009675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5356aebe-d543-4fd8-a974-b92ec16ed048-metrics-client-ca\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.009896 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.009765    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5356aebe-d543-4fd8-a974-b92ec16ed048-serving-certs-ca-bundle\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.010276 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.010227    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5356aebe-d543-4fd8-a974-b92ec16ed048-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.010451 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.010427    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5356aebe-d543-4fd8-a974-b92ec16ed048-metrics-client-ca\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.010526 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.010457    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5356aebe-d543-4fd8-a974-b92ec16ed048-serving-certs-ca-bundle\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.012482 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.012457    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-secret-telemeter-client\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.013120 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.013077    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-federate-client-tls\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.013404 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.013367    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.013668 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.013651    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5356aebe-d543-4fd8-a974-b92ec16ed048-telemeter-client-tls\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.018173 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.018130    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbsvh\" (UniqueName: \"kubernetes.io/projected/5356aebe-d543-4fd8-a974-b92ec16ed048-kube-api-access-rbsvh\") pod \"telemeter-client-7fd8dd5986-qw76j\" (UID: \"5356aebe-d543-4fd8-a974-b92ec16ed048\") " pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.148281 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.148244    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"
Jul 10 18:59:35.382442 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.382382    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fd8dd5986-qw76j"]
Jul 10 18:59:35.386650 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:35.386621    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5356aebe_d543_4fd8_a974_b92ec16ed048.slice/crio-5517fc414411305c9f33e06bd9a47e5c4c720f85ffc443b2dae1d67831152ba5 WatchSource:0}: Error finding container 5517fc414411305c9f33e06bd9a47e5c4c720f85ffc443b2dae1d67831152ba5: Status 404 returned error can't find the container with id 5517fc414411305c9f33e06bd9a47e5c4c720f85ffc443b2dae1d67831152ba5
Jul 10 18:59:35.863633 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.863604    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8668dbcd44-9w7qg"]
Jul 10 18:59:35.866307 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.866281    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:35.869083 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.869058    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 10 18:59:35.869193 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.869062    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 10 18:59:35.869193 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.869115    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 10 18:59:35.869484 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.869466    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 10 18:59:35.869834 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.869815    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 10 18:59:35.869834 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.869825    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rgkbr\""
Jul 10 18:59:35.889664 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.889639    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7" event={"ID":"b6c7a7f8-98f7-4469-a203-be4c8eb99869","Type":"ContainerStarted","Data":"20d5a99df1ffcacaa7ad5cc2dcb18457dd9ef387544449bc1a7d8c2ab3178909"}
Jul 10 18:59:35.889771 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.889723    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7" event={"ID":"b6c7a7f8-98f7-4469-a203-be4c8eb99869","Type":"ContainerStarted","Data":"b0ddd831490594394d19a9ae491cb7dc2d578241ff58194a3c1b750afe7dfc33"}
Jul 10 18:59:35.889846 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.889774    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:35.889846 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.889815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7" event={"ID":"b6c7a7f8-98f7-4469-a203-be4c8eb99869","Type":"ContainerStarted","Data":"edb3ce9a99c9e66b86e239f17ebbad79fc064db97166468b4126b95efbc1ccea"}
Jul 10 18:59:35.891058 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.891023    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j" event={"ID":"5356aebe-d543-4fd8-a974-b92ec16ed048","Type":"ContainerStarted","Data":"5517fc414411305c9f33e06bd9a47e5c4c720f85ffc443b2dae1d67831152ba5"}
Jul 10 18:59:35.894124 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.894103    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8668dbcd44-9w7qg"]
Jul 10 18:59:35.938715 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.938654    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7" podStartSLOduration=1.899335416 podStartE2EDuration="4.938634125s" totalImagesPullingTime="3.039298709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:35.935433248 +0000 UTC m=+79.187080580" watchObservedRunningTime="2026-07-10 18:59:35.938634125 +0000 UTC m=+79.190281455"
Jul 10 18:59:35.962876 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.962843    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:35.976690 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.974315    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:59:35.979355 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.979328    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:35.985486 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.985459    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 10 18:59:35.985601 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.985570    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 10 18:59:35.985694 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.985673    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vmnnj\""
Jul 10 18:59:35.985868 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.985838    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 10 18:59:35.986014 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.985965    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 10 18:59:35.986549 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.986145    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 10 18:59:35.986549 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.986233    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 10 18:59:35.986549 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.986312    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 10 18:59:35.986549 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.986355    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 10 18:59:35.986549 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.986452    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 10 18:59:35.986549 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.986528    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 10 18:59:35.987018 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.986536    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 10 18:59:35.987018 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.986706    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 10 18:59:35.987018 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.986838    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 10 18:59:35.987018 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.986928    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9mf7mepj7tkfi\""
Jul 10 18:59:35.987225 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.987046    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 10 18:59:35.989138 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:35.988330    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 10 18:59:36.002356 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.002329    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:59:36.022646 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.020317    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-config\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.022646 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.020392    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wfkr\" (UniqueName: \"kubernetes.io/projected/bdb82413-26a1-4d3a-a70f-62d92b1057fb-kube-api-access-8wfkr\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.022646 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.020509    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-oauth-config\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.022646 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.020578    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-oauth-serving-cert\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.022646 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.020633    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-serving-cert\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.022646 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.020658    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-service-ca\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.122115 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122115 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122115 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122114    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-oauth-serving-cert\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.122447 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122301    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122447 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122342    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122447 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-serving-cert\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.122601 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122459    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-service-ca\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.122601 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-config\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.122601 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122527    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122754 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122626    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122754 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122662    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122754 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122701    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wfkr\" (UniqueName: \"kubernetes.io/projected/bdb82413-26a1-4d3a-a70f-62d92b1057fb-kube-api-access-8wfkr\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.122896 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122729    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122896 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122896 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122803    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122896 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122839    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.122896 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122893    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-config\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.123179 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.122929    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.123179 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123008    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.123179 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123048    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.123179 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123071    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.123179 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-oauth-config\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.123179 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123149    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.123482 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123184    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.123482 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123216    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ntff\" (UniqueName: \"kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-kube-api-access-9ntff\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.123482 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123263    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.123663 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123639    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-service-ca\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.123961 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123934    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-config\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.123961 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.123951    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-oauth-serving-cert\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.126009 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.125989    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-oauth-config\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.126149 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.126129    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-serving-cert\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.131527 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.131504    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wfkr\" (UniqueName: \"kubernetes.io/projected/bdb82413-26a1-4d3a-a70f-62d92b1057fb-kube-api-access-8wfkr\") pod \"console-8668dbcd44-9w7qg\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") " pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.177296 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.177268    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.223948    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-config\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224001    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224073    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224099    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224149    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224203    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ntff\" (UniqueName: \"kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-kube-api-access-9ntff\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224228    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224261    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224286    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224322    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224350    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224393    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224426    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224458    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.224867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.225879 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224545    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.225879 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224571    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.225879 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.224594    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.229142 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.227316    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.229142 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.227619    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.229142 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.228523    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.229142 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.228799    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.229977 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.229643    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.231698 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.230609    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.231698 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.231443    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.233968 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.233945    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.234592 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.234564    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.236759 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.236619    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.237335 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.237036    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.237335 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.237231    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.237335 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.237252    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.237335 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.237297    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.237568 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.237501    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-config\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.237823 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.237728    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.237943 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.237860    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.240169 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.238753    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ntff\" (UniqueName: \"kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-kube-api-access-9ntff\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.240169 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.239219    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.240621 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.240535    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.292308 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.292007    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:36.357280 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.357238    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8668dbcd44-9w7qg"]
Jul 10 18:59:36.362190 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:36.362129    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb82413_26a1_4d3a_a70f_62d92b1057fb.slice/crio-de1ad60a2e47a970785767d40bd87f3b01763d512cdb507ee382683100b7b8fd WatchSource:0}: Error finding container de1ad60a2e47a970785767d40bd87f3b01763d512cdb507ee382683100b7b8fd: Status 404 returned error can't find the container with id de1ad60a2e47a970785767d40bd87f3b01763d512cdb507ee382683100b7b8fd
Jul 10 18:59:36.472397 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.472288    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:59:36.897886 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.897841    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8668dbcd44-9w7qg" event={"ID":"bdb82413-26a1-4d3a-a70f-62d92b1057fb","Type":"ContainerStarted","Data":"de1ad60a2e47a970785767d40bd87f3b01763d512cdb507ee382683100b7b8fd"}
Jul 10 18:59:36.899382 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:36.899347    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerStarted","Data":"5944fc9f29ca3af2ee88f3dae19938c849355e05ac45e0a3767d1c0ddb3a41b5"}
Jul 10 18:59:38.244917 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:38.244869    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:38.407072 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:38.407013    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:38.514141 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:38.514076    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:39.406058 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:39.405921    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:39.508376 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:39.508335    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:40.020597 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:40.020547    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8446f8d676-trjrq" podUID="a8dc518f-6220-4edc-bb98-4af59a9a65d5" containerName="registry" containerID="cri-o://a844c9f63544e1703f206c9fc4e8f04b9b0fcfa04dd644878275c0bf0a0906d7" gracePeriod=30
Jul 10 18:59:41.786232 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:41.786204    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:41.900351 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:41.900317    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:41.908431 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:41.908406    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77c964dcc5-89zt7"
Jul 10 18:59:42.010435 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:42.010395    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:44.157451 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:44.157419    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:44.227186 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:44.227145    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:44.998052 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:44.998016    2567 patch_prober.go:28] interesting pod/image-registry-8446f8d676-trjrq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.133.0.6:5000/healthz\": dial tcp 10.133.0.6:5000: connect: connection refused" start-of-body=
Jul 10 18:59:44.998242 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:44.998071    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8446f8d676-trjrq" podUID="a8dc518f-6220-4edc-bb98-4af59a9a65d5" containerName="registry" probeResult="failure" output="Get \"https://10.133.0.6:5000/healthz\": dial tcp 10.133.0.6:5000: connect: connection refused"
Jul 10 18:59:45.291411 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:45.291335    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:45.387415 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:45.387384    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:45.946673 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:45.946585    2567 generic.go:356] "Generic (PLEG): container finished" podID="a8dc518f-6220-4edc-bb98-4af59a9a65d5" containerID="a844c9f63544e1703f206c9fc4e8f04b9b0fcfa04dd644878275c0bf0a0906d7" exitCode=0
Jul 10 18:59:45.946813 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:45.946670    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8446f8d676-trjrq" event={"ID":"a8dc518f-6220-4edc-bb98-4af59a9a65d5","Type":"ContainerDied","Data":"a844c9f63544e1703f206c9fc4e8f04b9b0fcfa04dd644878275c0bf0a0906d7"}
Jul 10 18:59:47.027209 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.027182    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:59:47.124640 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.124615    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-trusted-ca\") pod \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") "
Jul 10 18:59:47.124640 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.124643    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-certificates\") pod \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") "
Jul 10 18:59:47.124840 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.124670    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls\") pod \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") "
Jul 10 18:59:47.124840 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.124697    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8dc518f-6220-4edc-bb98-4af59a9a65d5-ca-trust-extracted\") pod \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") "
Jul 10 18:59:47.124840 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.124726    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-bound-sa-token\") pod \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") "
Jul 10 18:59:47.124840 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.124797    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9572\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-kube-api-access-w9572\") pod \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") "
Jul 10 18:59:47.125070 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.124839    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-image-registry-private-configuration\") pod \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") "
Jul 10 18:59:47.125070 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.124869    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-installation-pull-secrets\") pod \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\" (UID: \"a8dc518f-6220-4edc-bb98-4af59a9a65d5\") "
Jul 10 18:59:47.125070 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.125049    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a8dc518f-6220-4edc-bb98-4af59a9a65d5" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:59:47.125235 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.125171    2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-certificates\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 18:59:47.125479 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.125448    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a8dc518f-6220-4edc-bb98-4af59a9a65d5" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:59:47.127417 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.127386    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-kube-api-access-w9572" (OuterVolumeSpecName: "kube-api-access-w9572") pod "a8dc518f-6220-4edc-bb98-4af59a9a65d5" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5"). InnerVolumeSpecName "kube-api-access-w9572". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:59:47.127585 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.127566    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a8dc518f-6220-4edc-bb98-4af59a9a65d5" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:59:47.127654 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.127598    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a8dc518f-6220-4edc-bb98-4af59a9a65d5" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:59:47.127703 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.127679    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a8dc518f-6220-4edc-bb98-4af59a9a65d5" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:59:47.127745 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.127706    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a8dc518f-6220-4edc-bb98-4af59a9a65d5" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:59:47.133688 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.133665    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8dc518f-6220-4edc-bb98-4af59a9a65d5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a8dc518f-6220-4edc-bb98-4af59a9a65d5" (UID: "a8dc518f-6220-4edc-bb98-4af59a9a65d5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:59:47.226285 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.226261    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9572\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-kube-api-access-w9572\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 18:59:47.226405 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.226289    2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-image-registry-private-configuration\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 18:59:47.226405 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.226304    2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8dc518f-6220-4edc-bb98-4af59a9a65d5-installation-pull-secrets\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 18:59:47.226405 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.226317    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc518f-6220-4edc-bb98-4af59a9a65d5-trusted-ca\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 18:59:47.226405 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.226329    2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-registry-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 18:59:47.226405 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.226341    2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8dc518f-6220-4edc-bb98-4af59a9a65d5-ca-trust-extracted\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 18:59:47.226405 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.226352    2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8dc518f-6220-4edc-bb98-4af59a9a65d5-bound-sa-token\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 18:59:47.454723 ip-10-0-136-15 kubenswrapper[2567]: E0710 18:59:47.454700    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dc518f_6220_4edc_bb98_4af59a9a65d5.slice/crio-85e25348437f78d04bc40397209af2f5cced064e1c6e54584619934a94cdb6ef\": RecentStats: unable to find data in memory cache]"
Jul 10 18:59:47.956867 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.956834    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8446f8d676-trjrq"
Jul 10 18:59:47.957081 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.956835    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8446f8d676-trjrq" event={"ID":"a8dc518f-6220-4edc-bb98-4af59a9a65d5","Type":"ContainerDied","Data":"85e25348437f78d04bc40397209af2f5cced064e1c6e54584619934a94cdb6ef"}
Jul 10 18:59:47.957081 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.956997    2567 scope.go:117] "RemoveContainer" containerID="a844c9f63544e1703f206c9fc4e8f04b9b0fcfa04dd644878275c0bf0a0906d7"
Jul 10 18:59:47.959478 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.959452    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-96gct" event={"ID":"116f558a-6e70-474a-83d8-fa124957af41","Type":"ContainerStarted","Data":"126bbc0a552cf4557ae72913a395f705f9ca39b581adde2640033e2849a7cb04"}
Jul 10 18:59:47.960133 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.960105    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-96gct"
Jul 10 18:59:47.962814 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.962790    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j" event={"ID":"5356aebe-d543-4fd8-a974-b92ec16ed048","Type":"ContainerStarted","Data":"9bca33aafd8556403fe13085d813d7342bd2472fd2edc5d931a651af2a5a63ec"}
Jul 10 18:59:47.962951 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.962846    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j" event={"ID":"5356aebe-d543-4fd8-a974-b92ec16ed048","Type":"ContainerStarted","Data":"1ad46e3892e427ff30993ac1c5e7eeb12d3d214e6ce1886afde825ff8ddcb419"}
Jul 10 18:59:47.962951 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.962866    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j" event={"ID":"5356aebe-d543-4fd8-a974-b92ec16ed048","Type":"ContainerStarted","Data":"c015b439e75c2d057b050a8150364f30901cdc27d77d9f61bd169b488e495a6a"}
Jul 10 18:59:47.964642 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.964616    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8668dbcd44-9w7qg" event={"ID":"bdb82413-26a1-4d3a-a70f-62d92b1057fb","Type":"ContainerStarted","Data":"751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c"}
Jul 10 18:59:47.966528 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.966502    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerID="1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4" exitCode=0
Jul 10 18:59:47.966639 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.966540    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerDied","Data":"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"}
Jul 10 18:59:47.977760 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.977728    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8446f8d676-trjrq"]
Jul 10 18:59:47.979735 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.979712    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-96gct"
Jul 10 18:59:47.982810 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:47.982775    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8446f8d676-trjrq"]
Jul 10 18:59:48.002040 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:48.001994    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-96gct" podStartSLOduration=3.36106093 podStartE2EDuration="23.001978698s" totalImagesPullingTime="19.640917768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:48.001491463 +0000 UTC m=+91.253138793" watchObservedRunningTime="2026-07-10 18:59:48.001978698 +0000 UTC m=+91.253626027"
Jul 10 18:59:48.050819 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:48.050697    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7fd8dd5986-qw76j" podStartSLOduration=2.29392389 podStartE2EDuration="14.050678907s" totalImagesPullingTime="11.756755017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:48.049656467 +0000 UTC m=+91.301303830" watchObservedRunningTime="2026-07-10 18:59:48.050678907 +0000 UTC m=+91.302326238"
Jul 10 18:59:48.078174 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:48.078126    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8668dbcd44-9w7qg" podStartSLOduration=2.289787918 podStartE2EDuration="13.078109149s" totalImagesPullingTime="10.788321231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:48.076562757 +0000 UTC m=+91.328210087" watchObservedRunningTime="2026-07-10 18:59:48.078109149 +0000 UTC m=+91.329756480"
Jul 10 18:59:49.420523 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:49.420493    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8dc518f-6220-4edc-bb98-4af59a9a65d5" path="/var/lib/kubelet/pods/a8dc518f-6220-4edc-bb98-4af59a9a65d5/volumes"
Jul 10 18:59:50.954212 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:50.954165    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:51.029958 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:51.029805    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:52.326178 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.326143    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8668dbcd44-9w7qg"]
Jul 10 18:59:52.429308 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.429283    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-95dd65648-d927f"]
Jul 10 18:59:52.429677 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.429657    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8dc518f-6220-4edc-bb98-4af59a9a65d5" containerName="registry"
Jul 10 18:59:52.429768 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.429679    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8dc518f-6220-4edc-bb98-4af59a9a65d5" containerName="registry"
Jul 10 18:59:52.429768 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.429744    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8dc518f-6220-4edc-bb98-4af59a9a65d5" containerName="registry"
Jul 10 18:59:52.458945 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.458919    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.467881 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.467853    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95dd65648-d927f"]
Jul 10 18:59:52.470206 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.469988    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 10 18:59:52.477111 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.477083    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czsgb\" (UniqueName: \"kubernetes.io/projected/dc530f93-06f0-4c7d-b810-df29185595bb-kube-api-access-czsgb\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.477221 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.477126    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-console-config\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.477221 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.477171    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-trusted-ca-bundle\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.477343 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.477224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-serving-cert\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.477343 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.477268    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-service-ca\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.477430 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.477336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-oauth-serving-cert\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.477480 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.477432    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-oauth-config\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.578343 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.578283    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czsgb\" (UniqueName: \"kubernetes.io/projected/dc530f93-06f0-4c7d-b810-df29185595bb-kube-api-access-czsgb\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.578457 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.578355    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-console-config\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.579102 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.579069    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-trusted-ca-bundle\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.579225 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.579129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-serving-cert\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.579225 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.579071    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-console-config\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.579348 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.579275    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-service-ca\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.579348 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.579306    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-oauth-serving-cert\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.579450 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.579377    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-oauth-config\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.580014 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.579988    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-trusted-ca-bundle\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.580106 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.580019    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-service-ca\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.580170 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.580151    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-oauth-serving-cert\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.582164 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.582139    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-serving-cert\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.582440 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.582421    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-oauth-config\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.594422 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.594398    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czsgb\" (UniqueName: \"kubernetes.io/projected/dc530f93-06f0-4c7d-b810-df29185595bb-kube-api-access-czsgb\") pod \"console-95dd65648-d927f\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") " pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.769878 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.769610    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95dd65648-d927f"
Jul 10 18:59:52.980093 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.980066    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95dd65648-d927f"]
Jul 10 18:59:52.983301 ip-10-0-136-15 kubenswrapper[2567]: W0710 18:59:52.983271    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc530f93_06f0_4c7d_b810_df29185595bb.slice/crio-20b91faf6ec4c3a39a0c3e07ac2601a83808455af3f8b0db4c29b3d9967dd032 WatchSource:0}: Error finding container 20b91faf6ec4c3a39a0c3e07ac2601a83808455af3f8b0db4c29b3d9967dd032: Status 404 returned error can't find the container with id 20b91faf6ec4c3a39a0c3e07ac2601a83808455af3f8b0db4c29b3d9967dd032
Jul 10 18:59:52.996777 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.996743    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerStarted","Data":"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"}
Jul 10 18:59:52.996901 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.996824    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerStarted","Data":"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"}
Jul 10 18:59:52.996901 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.996862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerStarted","Data":"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"}
Jul 10 18:59:52.996901 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.996893    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerStarted","Data":"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"}
Jul 10 18:59:52.998322 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:52.998284    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95dd65648-d927f" event={"ID":"dc530f93-06f0-4c7d-b810-df29185595bb","Type":"ContainerStarted","Data":"20b91faf6ec4c3a39a0c3e07ac2601a83808455af3f8b0db4c29b3d9967dd032"}
Jul 10 18:59:54.006243 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:54.006208    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerStarted","Data":"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"}
Jul 10 18:59:54.006803 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:54.006287    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerStarted","Data":"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"}
Jul 10 18:59:54.008132 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:54.008092    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95dd65648-d927f" event={"ID":"dc530f93-06f0-4c7d-b810-df29185595bb","Type":"ContainerStarted","Data":"80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1"}
Jul 10 18:59:54.059337 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:54.059275    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=12.839311707 podStartE2EDuration="19.059254012s" totalImagesPullingTime="6.206876251s" totalInitContainerRuntime="13.066054ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:54.056538303 +0000 UTC m=+97.308185658" watchObservedRunningTime="2026-07-10 18:59:54.059254012 +0000 UTC m=+97.310901342"
Jul 10 18:59:54.083440 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:54.083394    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-95dd65648-d927f" podStartSLOduration=2.083374122 podStartE2EDuration="2.083374122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:54.081617275 +0000 UTC m=+97.333264605" watchObservedRunningTime="2026-07-10 18:59:54.083374122 +0000 UTC m=+97.335021453"
Jul 10 18:59:54.449834 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:54.449798    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:54.543666 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:54.543629    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:55.587457 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:55.587423    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:56.178169 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:56.178136    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 18:59:56.292435 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:56.292399    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:59:56.348318 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:56.348287    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:56.439272 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:56.439186    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:56.846706 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:56.846675    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:56.948496 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:56.948471    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:57.140196 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:57.140127    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:57.216055 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:57.216029    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:58.848458 ip-10-0-136-15 kubenswrapper[2567]: I0710 18:59:58.848428    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mgkkh"
Jul 10 19:00:00.268442 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:00.268407    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:00.360924 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:00.360883    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:02.770570 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:02.770532    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-95dd65648-d927f"
Jul 10 19:00:02.771019 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:02.770611    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-95dd65648-d927f"
Jul 10 19:00:02.775342 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:02.775314    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-95dd65648-d927f"
Jul 10 19:00:03.048561 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:03.048482    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-95dd65648-d927f"
Jul 10 19:00:03.202379 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:03.202348    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:03.303145 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:03.303118    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:03.611035 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:03.610954    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:11.670011 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:11.669982    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:11.772522 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:11.772488    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:11.858720 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:11.858679    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:15.323201 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:15.323161    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:15.438975 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:15.438943    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:17.349269 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.349212    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8668dbcd44-9w7qg" podUID="bdb82413-26a1-4d3a-a70f-62d92b1057fb" containerName="console" containerID="cri-o://751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c" gracePeriod=15
Jul 10 19:00:17.622616 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.622586    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 19:00:17.787986 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.787878    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-oauth-serving-cert\") pod \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") "
Jul 10 19:00:17.787986 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.787928    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-service-ca\") pod \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") "
Jul 10 19:00:17.787986 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.787950    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-config\") pod \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") "
Jul 10 19:00:17.788259 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.788006    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-serving-cert\") pod \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") "
Jul 10 19:00:17.788259 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.788053    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wfkr\" (UniqueName: \"kubernetes.io/projected/bdb82413-26a1-4d3a-a70f-62d92b1057fb-kube-api-access-8wfkr\") pod \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") "
Jul 10 19:00:17.788259 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.788121    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-oauth-config\") pod \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\" (UID: \"bdb82413-26a1-4d3a-a70f-62d92b1057fb\") "
Jul 10 19:00:17.788412 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.788355    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bdb82413-26a1-4d3a-a70f-62d92b1057fb" (UID: "bdb82413-26a1-4d3a-a70f-62d92b1057fb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:17.788412 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.788371    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-service-ca" (OuterVolumeSpecName: "service-ca") pod "bdb82413-26a1-4d3a-a70f-62d92b1057fb" (UID: "bdb82413-26a1-4d3a-a70f-62d92b1057fb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:17.788514 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.788402    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-config" (OuterVolumeSpecName: "console-config") pod "bdb82413-26a1-4d3a-a70f-62d92b1057fb" (UID: "bdb82413-26a1-4d3a-a70f-62d92b1057fb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:17.790322 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.790289    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bdb82413-26a1-4d3a-a70f-62d92b1057fb" (UID: "bdb82413-26a1-4d3a-a70f-62d92b1057fb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:17.790448 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.790404    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb82413-26a1-4d3a-a70f-62d92b1057fb-kube-api-access-8wfkr" (OuterVolumeSpecName: "kube-api-access-8wfkr") pod "bdb82413-26a1-4d3a-a70f-62d92b1057fb" (UID: "bdb82413-26a1-4d3a-a70f-62d92b1057fb"). InnerVolumeSpecName "kube-api-access-8wfkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:17.790493 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.790476    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bdb82413-26a1-4d3a-a70f-62d92b1057fb" (UID: "bdb82413-26a1-4d3a-a70f-62d92b1057fb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:17.888953 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.888896    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:17.888953 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.888954    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-serving-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:17.889194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.888972    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wfkr\" (UniqueName: \"kubernetes.io/projected/bdb82413-26a1-4d3a-a70f-62d92b1057fb-kube-api-access-8wfkr\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:17.889194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.888985    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdb82413-26a1-4d3a-a70f-62d92b1057fb-console-oauth-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:17.889194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.888998    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-oauth-serving-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:17.889194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:17.889011    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb82413-26a1-4d3a-a70f-62d92b1057fb-service-ca\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:18.099590 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:18.099561    2567 generic.go:356] "Generic (PLEG): container finished" podID="bdb82413-26a1-4d3a-a70f-62d92b1057fb" containerID="751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c" exitCode=0
Jul 10 19:00:18.099765 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:18.099609    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8668dbcd44-9w7qg"
Jul 10 19:00:18.099765 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:18.099609    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8668dbcd44-9w7qg" event={"ID":"bdb82413-26a1-4d3a-a70f-62d92b1057fb","Type":"ContainerDied","Data":"751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c"}
Jul 10 19:00:18.099765 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:18.099661    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8668dbcd44-9w7qg" event={"ID":"bdb82413-26a1-4d3a-a70f-62d92b1057fb","Type":"ContainerDied","Data":"de1ad60a2e47a970785767d40bd87f3b01763d512cdb507ee382683100b7b8fd"}
Jul 10 19:00:18.099765 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:18.099682    2567 scope.go:117] "RemoveContainer" containerID="751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c"
Jul 10 19:00:18.108488 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:18.108471    2567 scope.go:117] "RemoveContainer" containerID="751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c"
Jul 10 19:00:18.108764 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:00:18.108744    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c\": container with ID starting with 751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c not found: ID does not exist" containerID="751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c"
Jul 10 19:00:18.108812 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:18.108773    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c"} err="failed to get container status \"751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c\": rpc error: code = NotFound desc = could not find container \"751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c\": container with ID starting with 751f12c38ccec8a80c48e868f1331c934475d22a2400eea0ce0c028cf6b9ee7c not found: ID does not exist"
Jul 10 19:00:18.121384 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:18.121363    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8668dbcd44-9w7qg"]
Jul 10 19:00:18.128759 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:18.128740    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8668dbcd44-9w7qg"]
Jul 10 19:00:19.418317 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:19.418286    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb82413-26a1-4d3a-a70f-62d92b1057fb" path="/var/lib/kubelet/pods/bdb82413-26a1-4d3a-a70f-62d92b1057fb/volumes"
Jul 10 19:00:20.015928 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:20.015885    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:36.293067 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:36.293037    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:36.345138 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:36.345113    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:37.183467 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:37.183441    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:45.948542 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:45.948474    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:46.035223 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:46.035195    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:54.599896 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.599855    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 19:00:54.601037 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.600972    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy-thanos" containerID="cri-o://174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a" gracePeriod=600
Jul 10 19:00:54.601506 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.601233    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy-web" containerID="cri-o://e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357" gracePeriod=600
Jul 10 19:00:54.601724 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.601273    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy" containerID="cri-o://33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51" gracePeriod=600
Jul 10 19:00:54.602323 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.601301    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="thanos-sidecar" containerID="cri-o://9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00" gracePeriod=600
Jul 10 19:00:54.602669 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.601318    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="config-reloader" containerID="cri-o://a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581" gracePeriod=600
Jul 10 19:00:54.603146 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.602522    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="prometheus" containerID="cri-o://5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe" gracePeriod=600
Jul 10 19:00:54.866423 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.866398    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:54.970902 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.970868    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-1\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971093 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.970934    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-config-out\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971093 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.970962    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-0\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971093 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971071    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-2\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971272 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971125    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-metrics-client-ca\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971272 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971159    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-metrics-client-certs\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971272 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971186    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-kubelet-serving-ca-bundle\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971272 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971217    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-web-config\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971272 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971257    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-thanos-prometheus-http-client-file\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971290    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-grpc-tls\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971322    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-config\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971349    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971254    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:54.971529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971398    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971429    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-db\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971453    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-tls\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971486    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-kube-rbac-proxy\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971514    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-serving-certs-ca-bundle\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971988 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971552    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-tls-assets\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971988 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971580    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-trusted-ca-bundle\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971988 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971605    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ntff\" (UniqueName: \"kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-kube-api-access-9ntff\") pod \"8e7aa1fc-5324-45a3-b38d-b50172a87192\" (UID: \"8e7aa1fc-5324-45a3-b38d-b50172a87192\") "
Jul 10 19:00:54.971988 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971705    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:54.971988 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971884    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:54.971988 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971921    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-metrics-client-ca\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:54.972630 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.971982    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:54.972630 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.972154    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:54.973311 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.973282    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:54.973658 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.973616    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:54.974768 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.974028    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:54.974768 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.974683    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:54.975503 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.975399    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:54.975503 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.975473    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:54.975503 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.975477    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-config-out" (OuterVolumeSpecName: "config-out") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:54.975838 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.975794    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:54.975965 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.975852    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-config" (OuterVolumeSpecName: "config") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:54.975965 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.975932    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:54.976125 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.976039    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-kube-api-access-9ntff" (OuterVolumeSpecName: "kube-api-access-9ntff") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "kube-api-access-9ntff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:54.976254 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.976226    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:54.976490 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.976474    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:54.976657 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.976636    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:54.977005 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.976977    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:54.984874 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:54.984854    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-web-config" (OuterVolumeSpecName: "web-config") pod "8e7aa1fc-5324-45a3-b38d-b50172a87192" (UID: "8e7aa1fc-5324-45a3-b38d-b50172a87192"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:55.072491 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072459    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072491 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072484    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072491 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072493    2567 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-metrics-client-certs\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072505    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072515    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-web-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072524    2567 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072533    2567 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-grpc-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072541    2567 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072550    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072559    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072568    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-k8s-db\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072576    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072584    2567 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e7aa1fc-5324-45a3-b38d-b50172a87192-secret-kube-rbac-proxy\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072592    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072600    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-tls-assets\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072608    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e7aa1fc-5324-45a3-b38d-b50172a87192-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072615    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ntff\" (UniqueName: \"kubernetes.io/projected/8e7aa1fc-5324-45a3-b38d-b50172a87192-kube-api-access-9ntff\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.072707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.072623    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e7aa1fc-5324-45a3-b38d-b50172a87192-config-out\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:00:55.240601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240516    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerID="174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a" exitCode=0
Jul 10 19:00:55.240601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240545    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerID="33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51" exitCode=0
Jul 10 19:00:55.240601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240552    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerID="e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357" exitCode=0
Jul 10 19:00:55.240601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240558    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerID="9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00" exitCode=0
Jul 10 19:00:55.240601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240565    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerID="a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581" exitCode=0
Jul 10 19:00:55.240601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240571    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerID="5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe" exitCode=0
Jul 10 19:00:55.240601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240589    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.240944 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240603    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerDied","Data":"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"}
Jul 10 19:00:55.240944 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240686    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerDied","Data":"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"}
Jul 10 19:00:55.240944 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240722    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerDied","Data":"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"}
Jul 10 19:00:55.240944 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240753    2567 scope.go:117] "RemoveContainer" containerID="174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"
Jul 10 19:00:55.240944 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240758    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerDied","Data":"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"}
Jul 10 19:00:55.240944 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240797    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerDied","Data":"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"}
Jul 10 19:00:55.240944 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240835    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerDied","Data":"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"}
Jul 10 19:00:55.240944 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.240876    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e7aa1fc-5324-45a3-b38d-b50172a87192","Type":"ContainerDied","Data":"5944fc9f29ca3af2ee88f3dae19938c849355e05ac45e0a3767d1c0ddb3a41b5"}
Jul 10 19:00:55.248727 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.248633    2567 scope.go:117] "RemoveContainer" containerID="33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"
Jul 10 19:00:55.255310 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.255286    2567 scope.go:117] "RemoveContainer" containerID="e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"
Jul 10 19:00:55.261699 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.261683    2567 scope.go:117] "RemoveContainer" containerID="9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"
Jul 10 19:00:55.265830 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.265810    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 19:00:55.268449 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.268433    2567 scope.go:117] "RemoveContainer" containerID="a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"
Jul 10 19:00:55.275346 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.275328    2567 scope.go:117] "RemoveContainer" containerID="5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"
Jul 10 19:00:55.275566 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.275547    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 19:00:55.282130 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.282115    2567 scope.go:117] "RemoveContainer" containerID="1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"
Jul 10 19:00:55.288352 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.288335    2567 scope.go:117] "RemoveContainer" containerID="174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"
Jul 10 19:00:55.288597 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:00:55.288580    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": container with ID starting with 174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a not found: ID does not exist" containerID="174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"
Jul 10 19:00:55.288656 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.288605    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"} err="failed to get container status \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": rpc error: code = NotFound desc = could not find container \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": container with ID starting with 174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a not found: ID does not exist"
Jul 10 19:00:55.288656 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.288622    2567 scope.go:117] "RemoveContainer" containerID="33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"
Jul 10 19:00:55.288855 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:00:55.288838    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": container with ID starting with 33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51 not found: ID does not exist" containerID="33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"
Jul 10 19:00:55.288894 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.288860    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"} err="failed to get container status \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": rpc error: code = NotFound desc = could not find container \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": container with ID starting with 33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51 not found: ID does not exist"
Jul 10 19:00:55.288894 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.288875    2567 scope.go:117] "RemoveContainer" containerID="e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"
Jul 10 19:00:55.289112 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:00:55.289099    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": container with ID starting with e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357 not found: ID does not exist" containerID="e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"
Jul 10 19:00:55.289151 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.289114    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"} err="failed to get container status \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": rpc error: code = NotFound desc = could not find container \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": container with ID starting with e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357 not found: ID does not exist"
Jul 10 19:00:55.289151 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.289125    2567 scope.go:117] "RemoveContainer" containerID="9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"
Jul 10 19:00:55.289348 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:00:55.289332    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": container with ID starting with 9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00 not found: ID does not exist" containerID="9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"
Jul 10 19:00:55.289384 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.289353    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"} err="failed to get container status \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": rpc error: code = NotFound desc = could not find container \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": container with ID starting with 9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00 not found: ID does not exist"
Jul 10 19:00:55.289384 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.289367    2567 scope.go:117] "RemoveContainer" containerID="a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"
Jul 10 19:00:55.289553 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:00:55.289539    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": container with ID starting with a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581 not found: ID does not exist" containerID="a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"
Jul 10 19:00:55.289588 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.289556    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"} err="failed to get container status \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": rpc error: code = NotFound desc = could not find container \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": container with ID starting with a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581 not found: ID does not exist"
Jul 10 19:00:55.289588 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.289568    2567 scope.go:117] "RemoveContainer" containerID="5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"
Jul 10 19:00:55.289746 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:00:55.289730    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": container with ID starting with 5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe not found: ID does not exist" containerID="5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"
Jul 10 19:00:55.289785 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.289748    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"} err="failed to get container status \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": rpc error: code = NotFound desc = could not find container \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": container with ID starting with 5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe not found: ID does not exist"
Jul 10 19:00:55.289785 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.289761    2567 scope.go:117] "RemoveContainer" containerID="1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"
Jul 10 19:00:55.289940 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:00:55.289926    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": container with ID starting with 1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4 not found: ID does not exist" containerID="1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"
Jul 10 19:00:55.289988 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.289943    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"} err="failed to get container status \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": rpc error: code = NotFound desc = could not find container \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": container with ID starting with 1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4 not found: ID does not exist"
Jul 10 19:00:55.289988 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.289955    2567 scope.go:117] "RemoveContainer" containerID="174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"
Jul 10 19:00:55.290174 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.290155    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"} err="failed to get container status \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": rpc error: code = NotFound desc = could not find container \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": container with ID starting with 174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a not found: ID does not exist"
Jul 10 19:00:55.290220 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.290176    2567 scope.go:117] "RemoveContainer" containerID="33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"
Jul 10 19:00:55.290374 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.290356    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"} err="failed to get container status \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": rpc error: code = NotFound desc = could not find container \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": container with ID starting with 33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51 not found: ID does not exist"
Jul 10 19:00:55.290412 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.290374    2567 scope.go:117] "RemoveContainer" containerID="e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"
Jul 10 19:00:55.290579 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.290560    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"} err="failed to get container status \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": rpc error: code = NotFound desc = could not find container \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": container with ID starting with e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357 not found: ID does not exist"
Jul 10 19:00:55.290634 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.290581    2567 scope.go:117] "RemoveContainer" containerID="9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"
Jul 10 19:00:55.290781 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.290763    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"} err="failed to get container status \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": rpc error: code = NotFound desc = could not find container \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": container with ID starting with 9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00 not found: ID does not exist"
Jul 10 19:00:55.290826 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.290780    2567 scope.go:117] "RemoveContainer" containerID="a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"
Jul 10 19:00:55.291021 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.291001    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"} err="failed to get container status \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": rpc error: code = NotFound desc = could not find container \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": container with ID starting with a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581 not found: ID does not exist"
Jul 10 19:00:55.291096 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.291023    2567 scope.go:117] "RemoveContainer" containerID="5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"
Jul 10 19:00:55.291254 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.291227    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"} err="failed to get container status \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": rpc error: code = NotFound desc = could not find container \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": container with ID starting with 5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe not found: ID does not exist"
Jul 10 19:00:55.291254 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.291254    2567 scope.go:117] "RemoveContainer" containerID="1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"
Jul 10 19:00:55.291486 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.291468    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"} err="failed to get container status \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": rpc error: code = NotFound desc = could not find container \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": container with ID starting with 1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4 not found: ID does not exist"
Jul 10 19:00:55.291533 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.291486    2567 scope.go:117] "RemoveContainer" containerID="174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"
Jul 10 19:00:55.291700 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.291682    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"} err="failed to get container status \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": rpc error: code = NotFound desc = could not find container \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": container with ID starting with 174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a not found: ID does not exist"
Jul 10 19:00:55.291768 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.291702    2567 scope.go:117] "RemoveContainer" containerID="33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"
Jul 10 19:00:55.291945 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.291900    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"} err="failed to get container status \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": rpc error: code = NotFound desc = could not find container \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": container with ID starting with 33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51 not found: ID does not exist"
Jul 10 19:00:55.291945 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.291944    2567 scope.go:117] "RemoveContainer" containerID="e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"
Jul 10 19:00:55.292165 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.292148    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"} err="failed to get container status \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": rpc error: code = NotFound desc = could not find container \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": container with ID starting with e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357 not found: ID does not exist"
Jul 10 19:00:55.292212 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.292165    2567 scope.go:117] "RemoveContainer" containerID="9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"
Jul 10 19:00:55.292362 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.292347    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"} err="failed to get container status \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": rpc error: code = NotFound desc = could not find container \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": container with ID starting with 9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00 not found: ID does not exist"
Jul 10 19:00:55.292403 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.292362    2567 scope.go:117] "RemoveContainer" containerID="a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"
Jul 10 19:00:55.292581 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.292560    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"} err="failed to get container status \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": rpc error: code = NotFound desc = could not find container \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": container with ID starting with a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581 not found: ID does not exist"
Jul 10 19:00:55.292621 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.292582    2567 scope.go:117] "RemoveContainer" containerID="5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"
Jul 10 19:00:55.292769 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.292755    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"} err="failed to get container status \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": rpc error: code = NotFound desc = could not find container \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": container with ID starting with 5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe not found: ID does not exist"
Jul 10 19:00:55.292805 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.292770    2567 scope.go:117] "RemoveContainer" containerID="1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"
Jul 10 19:00:55.292985 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.292968    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"} err="failed to get container status \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": rpc error: code = NotFound desc = could not find container \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": container with ID starting with 1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4 not found: ID does not exist"
Jul 10 19:00:55.293039 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.292985    2567 scope.go:117] "RemoveContainer" containerID="174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"
Jul 10 19:00:55.293144 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.293128    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"} err="failed to get container status \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": rpc error: code = NotFound desc = could not find container \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": container with ID starting with 174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a not found: ID does not exist"
Jul 10 19:00:55.293192 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.293144    2567 scope.go:117] "RemoveContainer" containerID="33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"
Jul 10 19:00:55.293348 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.293331    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"} err="failed to get container status \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": rpc error: code = NotFound desc = could not find container \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": container with ID starting with 33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51 not found: ID does not exist"
Jul 10 19:00:55.293348 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.293348    2567 scope.go:117] "RemoveContainer" containerID="e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"
Jul 10 19:00:55.293578 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.293561    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"} err="failed to get container status \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": rpc error: code = NotFound desc = could not find container \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": container with ID starting with e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357 not found: ID does not exist"
Jul 10 19:00:55.293625 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.293579    2567 scope.go:117] "RemoveContainer" containerID="9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"
Jul 10 19:00:55.293777 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.293762    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"} err="failed to get container status \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": rpc error: code = NotFound desc = could not find container \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": container with ID starting with 9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00 not found: ID does not exist"
Jul 10 19:00:55.293820 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.293778    2567 scope.go:117] "RemoveContainer" containerID="a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"
Jul 10 19:00:55.293987 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.293968    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"} err="failed to get container status \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": rpc error: code = NotFound desc = could not find container \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": container with ID starting with a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581 not found: ID does not exist"
Jul 10 19:00:55.294032 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.293987    2567 scope.go:117] "RemoveContainer" containerID="5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"
Jul 10 19:00:55.294210 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.294194    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"} err="failed to get container status \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": rpc error: code = NotFound desc = could not find container \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": container with ID starting with 5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe not found: ID does not exist"
Jul 10 19:00:55.294260 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.294210    2567 scope.go:117] "RemoveContainer" containerID="1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"
Jul 10 19:00:55.294436 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.294419    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"} err="failed to get container status \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": rpc error: code = NotFound desc = could not find container \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": container with ID starting with 1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4 not found: ID does not exist"
Jul 10 19:00:55.294476 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.294438    2567 scope.go:117] "RemoveContainer" containerID="174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"
Jul 10 19:00:55.294638 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.294617    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"} err="failed to get container status \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": rpc error: code = NotFound desc = could not find container \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": container with ID starting with 174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a not found: ID does not exist"
Jul 10 19:00:55.294680 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.294638    2567 scope.go:117] "RemoveContainer" containerID="33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"
Jul 10 19:00:55.294835 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.294816    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"} err="failed to get container status \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": rpc error: code = NotFound desc = could not find container \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": container with ID starting with 33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51 not found: ID does not exist"
Jul 10 19:00:55.294893 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.294834    2567 scope.go:117] "RemoveContainer" containerID="e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"
Jul 10 19:00:55.295067 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.295048    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"} err="failed to get container status \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": rpc error: code = NotFound desc = could not find container \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": container with ID starting with e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357 not found: ID does not exist"
Jul 10 19:00:55.295112 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.295070    2567 scope.go:117] "RemoveContainer" containerID="9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"
Jul 10 19:00:55.295284 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.295267    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"} err="failed to get container status \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": rpc error: code = NotFound desc = could not find container \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": container with ID starting with 9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00 not found: ID does not exist"
Jul 10 19:00:55.295333 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.295285    2567 scope.go:117] "RemoveContainer" containerID="a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"
Jul 10 19:00:55.295488 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.295473    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"} err="failed to get container status \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": rpc error: code = NotFound desc = could not find container \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": container with ID starting with a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581 not found: ID does not exist"
Jul 10 19:00:55.295538 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.295488    2567 scope.go:117] "RemoveContainer" containerID="5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"
Jul 10 19:00:55.295710 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.295690    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"} err="failed to get container status \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": rpc error: code = NotFound desc = could not find container \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": container with ID starting with 5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe not found: ID does not exist"
Jul 10 19:00:55.295774 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.295712    2567 scope.go:117] "RemoveContainer" containerID="1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"
Jul 10 19:00:55.295968 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.295895    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"} err="failed to get container status \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": rpc error: code = NotFound desc = could not find container \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": container with ID starting with 1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4 not found: ID does not exist"
Jul 10 19:00:55.295968 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.295930    2567 scope.go:117] "RemoveContainer" containerID="174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"
Jul 10 19:00:55.296128 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.296110    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a"} err="failed to get container status \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": rpc error: code = NotFound desc = could not find container \"174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a\": container with ID starting with 174b3a5e62ca83b69cb1c5dfa2749f8811f57986af66b6f1b22203e97b71896a not found: ID does not exist"
Jul 10 19:00:55.296192 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.296130    2567 scope.go:117] "RemoveContainer" containerID="33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"
Jul 10 19:00:55.296327 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.296312    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51"} err="failed to get container status \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": rpc error: code = NotFound desc = could not find container \"33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51\": container with ID starting with 33a95ed1d87de47ca5d1641615a04833f3455b4b9a9a2433261598567a233d51 not found: ID does not exist"
Jul 10 19:00:55.296389 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.296329    2567 scope.go:117] "RemoveContainer" containerID="e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"
Jul 10 19:00:55.296511 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.296496    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357"} err="failed to get container status \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": rpc error: code = NotFound desc = could not find container \"e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357\": container with ID starting with e7175bfceedaa2a03c24fe9025233430004b515698b0179e0ada0cb1e7d1a357 not found: ID does not exist"
Jul 10 19:00:55.296555 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.296510    2567 scope.go:117] "RemoveContainer" containerID="9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"
Jul 10 19:00:55.296713 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.296697    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00"} err="failed to get container status \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": rpc error: code = NotFound desc = could not find container \"9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00\": container with ID starting with 9830c4c8d8f6a4008f31320a63a6cfbcfd5981dab052c9b2f080cafb76055c00 not found: ID does not exist"
Jul 10 19:00:55.296776 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.296715    2567 scope.go:117] "RemoveContainer" containerID="a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"
Jul 10 19:00:55.296948 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.296931    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581"} err="failed to get container status \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": rpc error: code = NotFound desc = could not find container \"a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581\": container with ID starting with a6653733df6ca11e2c6c51d1a7b8163a7428e0e1c1e215238fa939b4f5437581 not found: ID does not exist"
Jul 10 19:00:55.296999 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.296949    2567 scope.go:117] "RemoveContainer" containerID="5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"
Jul 10 19:00:55.297168 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.297149    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe"} err="failed to get container status \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": rpc error: code = NotFound desc = could not find container \"5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe\": container with ID starting with 5e0948fd276f8d1f1af14be1b7a36784ba99cbeaee514b94c8b2f7c18262a4fe not found: ID does not exist"
Jul 10 19:00:55.297209 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.297169    2567 scope.go:117] "RemoveContainer" containerID="1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"
Jul 10 19:00:55.297336 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.297318    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4"} err="failed to get container status \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": rpc error: code = NotFound desc = could not find container \"1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4\": container with ID starting with 1317e55668cbffb4a3a414ddf45bf87945ff7b6818890aff7f93246cfb28dbe4 not found: ID does not exist"
Jul 10 19:00:55.308375 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308356    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 19:00:55.308619 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308606    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb82413-26a1-4d3a-a70f-62d92b1057fb" containerName="console"
Jul 10 19:00:55.308659 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308623    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb82413-26a1-4d3a-a70f-62d92b1057fb" containerName="console"
Jul 10 19:00:55.308659 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308636    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="config-reloader"
Jul 10 19:00:55.308659 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308642    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="config-reloader"
Jul 10 19:00:55.308659 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308651    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="thanos-sidecar"
Jul 10 19:00:55.308659 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308656    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="thanos-sidecar"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308664    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="prometheus"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308669    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="prometheus"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308676    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy-web"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308682    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy-web"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308687    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308694    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308707    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy-thanos"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308712    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy-thanos"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308721    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="init-config-reloader"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308726    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="init-config-reloader"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308771    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="config-reloader"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308782    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="prometheus"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308790    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="thanos-sidecar"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308795    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy-web"
Jul 10 19:00:55.308800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308800    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy-thanos"
Jul 10 19:00:55.309210 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308807    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb82413-26a1-4d3a-a70f-62d92b1057fb" containerName="console"
Jul 10 19:00:55.309210 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.308811    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" containerName="kube-rbac-proxy"
Jul 10 19:00:55.312264 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.312245    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.315095 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315076    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 10 19:00:55.315095 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315087    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 10 19:00:55.315390 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315375    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9mf7mepj7tkfi\""
Jul 10 19:00:55.315478 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315461    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 10 19:00:55.315549 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315483    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 10 19:00:55.315549 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315485    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 10 19:00:55.315646 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315570    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 10 19:00:55.315646 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315636    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 10 19:00:55.315867 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315852    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 10 19:00:55.315950 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315853    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 10 19:00:55.315950 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315926    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 10 19:00:55.316051 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.315970    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 10 19:00:55.316051 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.316009    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 10 19:00:55.317692 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.317674    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 10 19:00:55.319524 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.319506    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 10 19:00:55.323079 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.323063    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 10 19:00:55.324504 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.324491    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vmnnj\""
Jul 10 19:00:55.331698 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.331679    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 19:00:55.374112 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374215 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374121    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374215 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374145    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374325 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374325 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374325 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374304    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7j57\" (UniqueName: \"kubernetes.io/projected/c4c04f04-d1ad-4e70-b592-9572ff12d48d-kube-api-access-n7j57\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374414 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374333    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374414 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374359    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374414 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374378    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374414 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374395    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374557 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374418    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374557 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374443    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4c04f04-d1ad-4e70-b592-9572ff12d48d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374557 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374466    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-config\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374557 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374485    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374557 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374507    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374557 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374533    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374557 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374551    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374781 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374572    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374781 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374609    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.374781 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.374623    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4c04f04-d1ad-4e70-b592-9572ff12d48d-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.417491 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.417469    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7aa1fc-5324-45a3-b38d-b50172a87192" path="/var/lib/kubelet/pods/8e7aa1fc-5324-45a3-b38d-b50172a87192/volumes"
Jul 10 19:00:55.475293 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475263    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.475420 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475308    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.475420 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475332    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.475524 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475439    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.475524 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475467    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.475524 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475529    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4c04f04-d1ad-4e70-b592-9572ff12d48d-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475559    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475608    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475666    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7j57\" (UniqueName: \"kubernetes.io/projected/c4c04f04-d1ad-4e70-b592-9572ff12d48d-kube-api-access-n7j57\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475745    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475771    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475795    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475823    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475859    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475886    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4c04f04-d1ad-4e70-b592-9572ff12d48d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475938    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-config\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.475961    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476882 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.476180    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476882 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.476232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.476882 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.476742    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.477678 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.477654    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.478521 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.478496    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4c04f04-d1ad-4e70-b592-9572ff12d48d-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.478756 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.478721    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.478854 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.478732    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.479622 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.479147    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.479622 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.479259    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-config\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.479622 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.479457    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.479622 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.479512    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.479622 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.479512    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.480066 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.480043    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4c04f04-d1ad-4e70-b592-9572ff12d48d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.480299 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.480281    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.480953 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.480901    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4c04f04-d1ad-4e70-b592-9572ff12d48d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.481059 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.481046    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.481118 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.481075    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.481532 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.481515    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4c04f04-d1ad-4e70-b592-9572ff12d48d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.485066 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.485051    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7j57\" (UniqueName: \"kubernetes.io/projected/c4c04f04-d1ad-4e70-b592-9572ff12d48d-kube-api-access-n7j57\") pod \"prometheus-k8s-0\" (UID: \"c4c04f04-d1ad-4e70-b592-9572ff12d48d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.622432 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.622405    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:00:55.767662 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:55.767627    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 19:00:55.771633 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:00:55.771608    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c04f04_d1ad_4e70_b592_9572ff12d48d.slice/crio-5d82be304325d3a0363e644ab3e7d17a0ac5db63f02e987d945575c88ae85fc8 WatchSource:0}: Error finding container 5d82be304325d3a0363e644ab3e7d17a0ac5db63f02e987d945575c88ae85fc8: Status 404 returned error can't find the container with id 5d82be304325d3a0363e644ab3e7d17a0ac5db63f02e987d945575c88ae85fc8
Jul 10 19:00:56.245700 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:56.245668    2567 generic.go:356] "Generic (PLEG): container finished" podID="c4c04f04-d1ad-4e70-b592-9572ff12d48d" containerID="bc66dd9bc31d310dd9710688c5ab20aa27f7f532a19b44dc1252a45ba052b3e4" exitCode=0
Jul 10 19:00:56.245886 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:56.245761    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4c04f04-d1ad-4e70-b592-9572ff12d48d","Type":"ContainerDied","Data":"bc66dd9bc31d310dd9710688c5ab20aa27f7f532a19b44dc1252a45ba052b3e4"}
Jul 10 19:00:56.245886 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:56.245845    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4c04f04-d1ad-4e70-b592-9572ff12d48d","Type":"ContainerStarted","Data":"5d82be304325d3a0363e644ab3e7d17a0ac5db63f02e987d945575c88ae85fc8"}
Jul 10 19:00:57.253927 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:57.253883    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4c04f04-d1ad-4e70-b592-9572ff12d48d","Type":"ContainerStarted","Data":"8a0488ac140795b055cbdc1ebe3b8e4fea99f7eb6911b9dc032a295f8ab49183"}
Jul 10 19:00:57.254264 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:57.253960    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4c04f04-d1ad-4e70-b592-9572ff12d48d","Type":"ContainerStarted","Data":"b254bee0b769e94ed96c3bd0d8f9b0cc8551d46660c952a30ef91c28ae10bca3"}
Jul 10 19:00:57.254264 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:57.253986    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4c04f04-d1ad-4e70-b592-9572ff12d48d","Type":"ContainerStarted","Data":"c36b81e7bf4791a1ab53687248d40d0ae04a6ba5098227ebd1e0c69bb332da68"}
Jul 10 19:00:57.254264 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:57.254011    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4c04f04-d1ad-4e70-b592-9572ff12d48d","Type":"ContainerStarted","Data":"8ce1365c3707feafacb948490ed54b4e1139d2e27c8887989b15ab319536a262"}
Jul 10 19:00:57.254264 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:57.254037    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4c04f04-d1ad-4e70-b592-9572ff12d48d","Type":"ContainerStarted","Data":"4f82f3049d6c4ee1f72e32a08c983c7ce6d7cf435c7664b5d547bc9193ef8b3e"}
Jul 10 19:00:57.254264 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:57.254059    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4c04f04-d1ad-4e70-b592-9572ff12d48d","Type":"ContainerStarted","Data":"105fb4e4d3dac270b4aed77ce9ec4efbc401e76684750a61d11450c9bdf0cffb"}
Jul 10 19:00:57.284210 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:57.284155    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.261841944 podStartE2EDuration="2.284136056s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.294112ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:57.281751352 +0000 UTC m=+160.533398681" watchObservedRunningTime="2026-07-10 19:00:57.284136056 +0000 UTC m=+160.535783385"
Jul 10 19:00:57.862834 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:57.862799    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:57.964217 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:00:57.964181    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:00.622689 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:00.622655    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:01:00.991062 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:00.990982    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:01.049256 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:01.049217    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:11.563516 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:11.563433    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95dd65648-d927f"]
Jul 10 19:01:11.685568 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:11.685520    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:11.786457 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:11.786425    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:11.886304 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:11.886205    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:15.261294 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:15.261251    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:15.361430 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:15.361405    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:15.416097 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:15.416061    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:18.126575 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:18.126530    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:18.210034 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:18.209989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:19.218537 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:19.218500    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:36.584053 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:36.583994    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-95dd65648-d927f" podUID="dc530f93-06f0-4c7d-b810-df29185595bb" containerName="console" containerID="cri-o://80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1" gracePeriod=15
Jul 10 19:01:36.828957 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:36.828936    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95dd65648-d927f"
Jul 10 19:01:37.006628 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.006528    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-trusted-ca-bundle\") pod \"dc530f93-06f0-4c7d-b810-df29185595bb\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") "
Jul 10 19:01:37.006628 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.006582    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-oauth-config\") pod \"dc530f93-06f0-4c7d-b810-df29185595bb\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") "
Jul 10 19:01:37.006859 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.006677    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-oauth-serving-cert\") pod \"dc530f93-06f0-4c7d-b810-df29185595bb\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") "
Jul 10 19:01:37.006859 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.006745    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-service-ca\") pod \"dc530f93-06f0-4c7d-b810-df29185595bb\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") "
Jul 10 19:01:37.006859 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.006781    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-console-config\") pod \"dc530f93-06f0-4c7d-b810-df29185595bb\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") "
Jul 10 19:01:37.006859 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.006815    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-serving-cert\") pod \"dc530f93-06f0-4c7d-b810-df29185595bb\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") "
Jul 10 19:01:37.006859 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.006858    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czsgb\" (UniqueName: \"kubernetes.io/projected/dc530f93-06f0-4c7d-b810-df29185595bb-kube-api-access-czsgb\") pod \"dc530f93-06f0-4c7d-b810-df29185595bb\" (UID: \"dc530f93-06f0-4c7d-b810-df29185595bb\") "
Jul 10 19:01:37.007144 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.007105    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dc530f93-06f0-4c7d-b810-df29185595bb" (UID: "dc530f93-06f0-4c7d-b810-df29185595bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:01:37.007144 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.007112    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dc530f93-06f0-4c7d-b810-df29185595bb" (UID: "dc530f93-06f0-4c7d-b810-df29185595bb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:01:37.007249 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.007175    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "dc530f93-06f0-4c7d-b810-df29185595bb" (UID: "dc530f93-06f0-4c7d-b810-df29185595bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:01:37.007249 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.007181    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-console-config" (OuterVolumeSpecName: "console-config") pod "dc530f93-06f0-4c7d-b810-df29185595bb" (UID: "dc530f93-06f0-4c7d-b810-df29185595bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:01:37.008880 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.008846    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dc530f93-06f0-4c7d-b810-df29185595bb" (UID: "dc530f93-06f0-4c7d-b810-df29185595bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:01:37.009006 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.008887    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc530f93-06f0-4c7d-b810-df29185595bb-kube-api-access-czsgb" (OuterVolumeSpecName: "kube-api-access-czsgb") pod "dc530f93-06f0-4c7d-b810-df29185595bb" (UID: "dc530f93-06f0-4c7d-b810-df29185595bb"). InnerVolumeSpecName "kube-api-access-czsgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:01:37.009006 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.008927    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dc530f93-06f0-4c7d-b810-df29185595bb" (UID: "dc530f93-06f0-4c7d-b810-df29185595bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:01:37.107940 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.107892    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-service-ca\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:01:37.107940 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.107938    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-console-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:01:37.108130 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.107953    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-serving-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:01:37.108130 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.107968    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czsgb\" (UniqueName: \"kubernetes.io/projected/dc530f93-06f0-4c7d-b810-df29185595bb-kube-api-access-czsgb\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:01:37.108130 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.107981    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-trusted-ca-bundle\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:01:37.108130 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.107993    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc530f93-06f0-4c7d-b810-df29185595bb-console-oauth-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:01:37.108130 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.108007    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc530f93-06f0-4c7d-b810-df29185595bb-oauth-serving-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:01:37.396584 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.396550    2567 generic.go:356] "Generic (PLEG): container finished" podID="dc530f93-06f0-4c7d-b810-df29185595bb" containerID="80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1" exitCode=0
Jul 10 19:01:37.396758 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.396591    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95dd65648-d927f"
Jul 10 19:01:37.396758 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.396634    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95dd65648-d927f" event={"ID":"dc530f93-06f0-4c7d-b810-df29185595bb","Type":"ContainerDied","Data":"80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1"}
Jul 10 19:01:37.396758 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.396681    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95dd65648-d927f" event={"ID":"dc530f93-06f0-4c7d-b810-df29185595bb","Type":"ContainerDied","Data":"20b91faf6ec4c3a39a0c3e07ac2601a83808455af3f8b0db4c29b3d9967dd032"}
Jul 10 19:01:37.396758 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.396700    2567 scope.go:117] "RemoveContainer" containerID="80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1"
Jul 10 19:01:37.405139 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.405125    2567 scope.go:117] "RemoveContainer" containerID="80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1"
Jul 10 19:01:37.405404 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:01:37.405383    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1\": container with ID starting with 80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1 not found: ID does not exist" containerID="80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1"
Jul 10 19:01:37.405447 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.405414    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1"} err="failed to get container status \"80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1\": rpc error: code = NotFound desc = could not find container \"80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1\": container with ID starting with 80df9e0b71a0ef6d952d0a14e7bf6e5a93f4a4d962fc4b181529b8ae914910b1 not found: ID does not exist"
Jul 10 19:01:37.417523 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.417504    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95dd65648-d927f"]
Jul 10 19:01:37.423630 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:37.423606    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-95dd65648-d927f"]
Jul 10 19:01:39.418323 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:39.418284    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc530f93-06f0-4c7d-b810-df29185595bb" path="/var/lib/kubelet/pods/dc530f93-06f0-4c7d-b810-df29185595bb/volumes"
Jul 10 19:01:47.125972 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:47.125930    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:47.185971 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:47.185937    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:55.622950 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:55.622902    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:01:55.637847 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:55.637823    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:01:56.477204 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:01:56.477180    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 19:02:02.058409 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:02:02.058367    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:02.149800 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:02:02.149767    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:02.209671 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:02:02.209636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:27.162838 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:02:27.162802    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:32.828896 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:02:32.828861    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:32.883988 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:02:32.883961    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:33.209148 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:02:33.209075    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:33.308966 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:02:33.308942    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:33.367115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:02:33.367091    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:03:15.616419 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:03:15.616379    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:03:16.977778 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:03:16.977738    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:03:17.187364 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:03:17.187329    2567 kubelet.go:1648] "Image garbage collection succeeded"
Jul 10 19:03:19.567290 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:03:19.567245    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:03:19.631811 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:03:19.631782    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:08.194055 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:08.194020    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:08.286611 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:08.286579    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:13.808935 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:13.808813    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:14.129950 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:14.129858    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:14.569816 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:14.569783    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:14.768639 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:14.768609    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:14.914329 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:14.914250    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:15.089873 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:15.089840    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:15.182604 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:15.182475    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:15.419044 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:15.418992    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:15.518602 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:15.518488    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:15.631603 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:15.631564    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:15.819946 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:15.819892    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:15.897163 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:15.897122    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:35.963796 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:35.963760    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:54.630534 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.630500    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d4ccf654f-7v6t7"]
Jul 10 19:04:54.631018 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.630782    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc530f93-06f0-4c7d-b810-df29185595bb" containerName="console"
Jul 10 19:04:54.631018 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.630795    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc530f93-06f0-4c7d-b810-df29185595bb" containerName="console"
Jul 10 19:04:54.631018 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.630851    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc530f93-06f0-4c7d-b810-df29185595bb" containerName="console"
Jul 10 19:04:54.633779 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.633760    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.636290 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.636265    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 10 19:04:54.637352 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.637335    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rgkbr\""
Jul 10 19:04:54.637483 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.637465    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 10 19:04:54.637541 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.637477    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 10 19:04:54.637541 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.637492    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 10 19:04:54.637823 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.637807    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 10 19:04:54.650335 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.650311    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4ccf654f-7v6t7"]
Jul 10 19:04:54.652304 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.652287    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 10 19:04:54.741833 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.741811    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:04:54.742944 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.742921    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z567t\" (UniqueName: \"kubernetes.io/projected/98284170-bdd0-4e98-bd5c-cb3d28fe396c-kube-api-access-z567t\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.743044 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.742965    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98284170-bdd0-4e98-bd5c-cb3d28fe396c-console-oauth-config\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.743044 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.743009    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98284170-bdd0-4e98-bd5c-cb3d28fe396c-console-serving-cert\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.743110 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.743041    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-console-config\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.743110 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.743086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-oauth-serving-cert\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.743173 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.743109    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-service-ca\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.743173 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.743157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-trusted-ca-bundle\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.844335 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.844306    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98284170-bdd0-4e98-bd5c-cb3d28fe396c-console-serving-cert\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.844413 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.844344    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-console-config\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.844413 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.844394    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-oauth-serving-cert\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.844478 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.844421    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-service-ca\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.844478 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.844445    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-trusted-ca-bundle\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.844550 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.844476    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z567t\" (UniqueName: \"kubernetes.io/projected/98284170-bdd0-4e98-bd5c-cb3d28fe396c-kube-api-access-z567t\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.844635 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.844609    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98284170-bdd0-4e98-bd5c-cb3d28fe396c-console-oauth-config\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.845293 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.845269    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-oauth-serving-cert\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.845409 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.845269    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-console-config\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.845409 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.845351    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-service-ca\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.845409 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.845397    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98284170-bdd0-4e98-bd5c-cb3d28fe396c-trusted-ca-bundle\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.846720 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.846693    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98284170-bdd0-4e98-bd5c-cb3d28fe396c-console-serving-cert\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.847000 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.846982    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98284170-bdd0-4e98-bd5c-cb3d28fe396c-console-oauth-config\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.853060 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.853037    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z567t\" (UniqueName: \"kubernetes.io/projected/98284170-bdd0-4e98-bd5c-cb3d28fe396c-kube-api-access-z567t\") pod \"console-5d4ccf654f-7v6t7\" (UID: \"98284170-bdd0-4e98-bd5c-cb3d28fe396c\") " pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:54.943079 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:54.942998    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:04:55.074561 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:55.074533    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4ccf654f-7v6t7"]
Jul 10 19:04:55.077717 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:04:55.077688    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98284170_bdd0_4e98_bd5c_cb3d28fe396c.slice/crio-e32ac5cf261ed8af7db26205f0df0db4e549335b2f1b74afbd701167aae57bce WatchSource:0}: Error finding container e32ac5cf261ed8af7db26205f0df0db4e549335b2f1b74afbd701167aae57bce: Status 404 returned error can't find the container with id e32ac5cf261ed8af7db26205f0df0db4e549335b2f1b74afbd701167aae57bce
Jul 10 19:04:56.088825 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:56.088789    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4ccf654f-7v6t7" event={"ID":"98284170-bdd0-4e98-bd5c-cb3d28fe396c","Type":"ContainerStarted","Data":"601adaed8df73f8c453a67c8a668a1c6f8c291883056e43ae19d211cfd5a85b0"}
Jul 10 19:04:56.089317 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:56.088876    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4ccf654f-7v6t7" event={"ID":"98284170-bdd0-4e98-bd5c-cb3d28fe396c","Type":"ContainerStarted","Data":"e32ac5cf261ed8af7db26205f0df0db4e549335b2f1b74afbd701167aae57bce"}
Jul 10 19:04:56.109237 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:04:56.109190    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d4ccf654f-7v6t7" podStartSLOduration=2.109175688 podStartE2EDuration="2.109175688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:56.108207668 +0000 UTC m=+399.359855014" watchObservedRunningTime="2026-07-10 19:04:56.109175688 +0000 UTC m=+399.360823017"
Jul 10 19:05:04.944108 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:04.944054    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:05:04.944512 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:04.944136    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:05:04.948762 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:04.948739    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:05:05.125343 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:05.125315    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d4ccf654f-7v6t7"
Jul 10 19:05:05.281978 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:05.281867    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:05:05.344889 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:05.344840    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:05:05.402130 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:05.402097    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:05:30.281136 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.281102    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:05:30.424297 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.424246    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:05:30.444445 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.444414    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-98r84"]
Jul 10 19:05:30.447808 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.447791    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-98r84"
Jul 10 19:05:30.455696 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.455674    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 10 19:05:30.456647 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.456628    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-74dzv\""
Jul 10 19:05:30.459018 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.459000    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 10 19:05:30.459114 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.459040    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 10 19:05:30.474439 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.474415    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-98r84"]
Jul 10 19:05:30.524739 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.524710    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f744953-d7b3-45a0-8268-e4e75c7c2feb-cert\") pod \"llmisvc-controller-manager-654c557856-98r84\" (UID: \"6f744953-d7b3-45a0-8268-e4e75c7c2feb\") " pod="kserve/llmisvc-controller-manager-654c557856-98r84"
Jul 10 19:05:30.524861 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.524775    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tgg4\" (UniqueName: \"kubernetes.io/projected/6f744953-d7b3-45a0-8268-e4e75c7c2feb-kube-api-access-5tgg4\") pod \"llmisvc-controller-manager-654c557856-98r84\" (UID: \"6f744953-d7b3-45a0-8268-e4e75c7c2feb\") " pod="kserve/llmisvc-controller-manager-654c557856-98r84"
Jul 10 19:05:30.626182 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.626150    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tgg4\" (UniqueName: \"kubernetes.io/projected/6f744953-d7b3-45a0-8268-e4e75c7c2feb-kube-api-access-5tgg4\") pod \"llmisvc-controller-manager-654c557856-98r84\" (UID: \"6f744953-d7b3-45a0-8268-e4e75c7c2feb\") " pod="kserve/llmisvc-controller-manager-654c557856-98r84"
Jul 10 19:05:30.626350 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.626205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f744953-d7b3-45a0-8268-e4e75c7c2feb-cert\") pod \"llmisvc-controller-manager-654c557856-98r84\" (UID: \"6f744953-d7b3-45a0-8268-e4e75c7c2feb\") " pod="kserve/llmisvc-controller-manager-654c557856-98r84"
Jul 10 19:05:30.628434 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.628407    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f744953-d7b3-45a0-8268-e4e75c7c2feb-cert\") pod \"llmisvc-controller-manager-654c557856-98r84\" (UID: \"6f744953-d7b3-45a0-8268-e4e75c7c2feb\") " pod="kserve/llmisvc-controller-manager-654c557856-98r84"
Jul 10 19:05:30.640751 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.640728    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:05:30.646014 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.645992    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tgg4\" (UniqueName: \"kubernetes.io/projected/6f744953-d7b3-45a0-8268-e4e75c7c2feb-kube-api-access-5tgg4\") pod \"llmisvc-controller-manager-654c557856-98r84\" (UID: \"6f744953-d7b3-45a0-8268-e4e75c7c2feb\") " pod="kserve/llmisvc-controller-manager-654c557856-98r84"
Jul 10 19:05:30.758033 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.758002    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-98r84"
Jul 10 19:05:30.907580 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.907467    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-98r84"]
Jul 10 19:05:30.910184 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:05:30.910158    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f744953_d7b3_45a0_8268_e4e75c7c2feb.slice/crio-c3f87e5ae73c3d38e2629ab90492c60576c7cdde0f7bcab5f7e5d181b735742c WatchSource:0}: Error finding container c3f87e5ae73c3d38e2629ab90492c60576c7cdde0f7bcab5f7e5d181b735742c: Status 404 returned error can't find the container with id c3f87e5ae73c3d38e2629ab90492c60576c7cdde0f7bcab5f7e5d181b735742c
Jul 10 19:05:30.911622 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.911602    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:05:30.968759 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:30.968735    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:05:31.216643 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:31.216567    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-98r84" event={"ID":"6f744953-d7b3-45a0-8268-e4e75c7c2feb","Type":"ContainerStarted","Data":"c3f87e5ae73c3d38e2629ab90492c60576c7cdde0f7bcab5f7e5d181b735742c"}
Jul 10 19:05:31.447089 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:31.447050    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:05:31.550712 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:31.550673    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:05:34.228766 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:34.228674    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-98r84" event={"ID":"6f744953-d7b3-45a0-8268-e4e75c7c2feb","Type":"ContainerStarted","Data":"b549f1226877af15b264a8d3e7a0fbf6408567bae1b9f4ac3833279939ed776d"}
Jul 10 19:05:34.229175 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:05:34.228788    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-98r84"
Jul 10 19:06:05.234141 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:05.234110    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-98r84"
Jul 10 19:06:05.250957 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:05.250890    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-98r84" podStartSLOduration=32.305277198 podStartE2EDuration="35.25087611s" totalImagesPullingTime="2.945598912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:34.255307438 +0000 UTC m=+437.506954767" watchObservedRunningTime="2026-07-10 19:06:05.25087611 +0000 UTC m=+468.502523439"
Jul 10 19:06:41.130518 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.130482    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:06:41.198838 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.198807    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-lqj7q"]
Jul 10 19:06:41.201750 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.201734    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-lqj7q"
Jul 10 19:06:41.204478 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.204454    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-xppd8\""
Jul 10 19:06:41.204700 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.204685    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 10 19:06:41.214672 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.214647    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-lqj7q"]
Jul 10 19:06:41.236501 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.236475    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:06:41.275361 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.275339    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7hqz\" (UniqueName: \"kubernetes.io/projected/617f3522-3d9d-4800-9461-4dc7628bf8c5-kube-api-access-n7hqz\") pod \"odh-model-controller-84b54b7f97-lqj7q\" (UID: \"617f3522-3d9d-4800-9461-4dc7628bf8c5\") " pod="kserve/odh-model-controller-84b54b7f97-lqj7q"
Jul 10 19:06:41.275477 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.275399    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/617f3522-3d9d-4800-9461-4dc7628bf8c5-cert\") pod \"odh-model-controller-84b54b7f97-lqj7q\" (UID: \"617f3522-3d9d-4800-9461-4dc7628bf8c5\") " pod="kserve/odh-model-controller-84b54b7f97-lqj7q"
Jul 10 19:06:41.376142 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.376112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7hqz\" (UniqueName: \"kubernetes.io/projected/617f3522-3d9d-4800-9461-4dc7628bf8c5-kube-api-access-n7hqz\") pod \"odh-model-controller-84b54b7f97-lqj7q\" (UID: \"617f3522-3d9d-4800-9461-4dc7628bf8c5\") " pod="kserve/odh-model-controller-84b54b7f97-lqj7q"
Jul 10 19:06:41.376269 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.376161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/617f3522-3d9d-4800-9461-4dc7628bf8c5-cert\") pod \"odh-model-controller-84b54b7f97-lqj7q\" (UID: \"617f3522-3d9d-4800-9461-4dc7628bf8c5\") " pod="kserve/odh-model-controller-84b54b7f97-lqj7q"
Jul 10 19:06:41.378487 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.378469    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/617f3522-3d9d-4800-9461-4dc7628bf8c5-cert\") pod \"odh-model-controller-84b54b7f97-lqj7q\" (UID: \"617f3522-3d9d-4800-9461-4dc7628bf8c5\") " pod="kserve/odh-model-controller-84b54b7f97-lqj7q"
Jul 10 19:06:41.382276 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.382217    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:06:41.385158 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.385134    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7hqz\" (UniqueName: \"kubernetes.io/projected/617f3522-3d9d-4800-9461-4dc7628bf8c5-kube-api-access-n7hqz\") pod \"odh-model-controller-84b54b7f97-lqj7q\" (UID: \"617f3522-3d9d-4800-9461-4dc7628bf8c5\") " pod="kserve/odh-model-controller-84b54b7f97-lqj7q"
Jul 10 19:06:41.514154 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.514125    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-lqj7q"
Jul 10 19:06:41.558722 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.558685    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:06:41.639381 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.639307    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-lqj7q"]
Jul 10 19:06:41.642541 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:06:41.642522    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617f3522_3d9d_4800_9461_4dc7628bf8c5.slice/crio-2922c7336412fc4f24feb5702619d54bf9a23b88ab549ad397d050e538eb77e4 WatchSource:0}: Error finding container 2922c7336412fc4f24feb5702619d54bf9a23b88ab549ad397d050e538eb77e4: Status 404 returned error can't find the container with id 2922c7336412fc4f24feb5702619d54bf9a23b88ab549ad397d050e538eb77e4
Jul 10 19:06:41.929368 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:41.929291    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:06:42.002551 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:42.002515    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:06:42.468301 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:42.468267    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-lqj7q" event={"ID":"617f3522-3d9d-4800-9461-4dc7628bf8c5","Type":"ContainerStarted","Data":"2922c7336412fc4f24feb5702619d54bf9a23b88ab549ad397d050e538eb77e4"}
Jul 10 19:06:44.483059 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:44.483013    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-lqj7q" event={"ID":"617f3522-3d9d-4800-9461-4dc7628bf8c5","Type":"ContainerStarted","Data":"c331975b9245354e06b23c4a87e64c8355fddb56e3b6ff0a8ab640e55c73a0ad"}
Jul 10 19:06:44.483496 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:44.483144    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-lqj7q"
Jul 10 19:06:44.509516 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:44.509470    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-lqj7q" podStartSLOduration=0.793936717 podStartE2EDuration="3.509456243s" totalImagesPullingTime="2.715519526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:44.508816728 +0000 UTC m=+507.760464056" watchObservedRunningTime="2026-07-10 19:06:44.509456243 +0000 UTC m=+507.761103594"
Jul 10 19:06:55.489635 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:55.489608    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-lqj7q"
Jul 10 19:06:56.357898 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.357861    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-xbg6x"]
Jul 10 19:06:56.361303 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.361287    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xbg6x"
Jul 10 19:06:56.363899 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.363878    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 10 19:06:56.363899 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.363892    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-msbqc\""
Jul 10 19:06:56.370794 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.370770    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-xbg6x"]
Jul 10 19:06:56.496423 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.496389    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-model-cache\") pod \"s3-init-xbg6x\" (UID: \"3f4ef870-3dc2-46e0-960a-ba52569f9ee9\") " pod="kserve/s3-init-xbg6x"
Jul 10 19:06:56.496798 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.496503    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt4c7\" (UniqueName: \"kubernetes.io/projected/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-kube-api-access-mt4c7\") pod \"s3-init-xbg6x\" (UID: \"3f4ef870-3dc2-46e0-960a-ba52569f9ee9\") " pod="kserve/s3-init-xbg6x"
Jul 10 19:06:56.597461 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.597428    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt4c7\" (UniqueName: \"kubernetes.io/projected/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-kube-api-access-mt4c7\") pod \"s3-init-xbg6x\" (UID: \"3f4ef870-3dc2-46e0-960a-ba52569f9ee9\") " pod="kserve/s3-init-xbg6x"
Jul 10 19:06:56.597625 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.597482    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-model-cache\") pod \"s3-init-xbg6x\" (UID: \"3f4ef870-3dc2-46e0-960a-ba52569f9ee9\") " pod="kserve/s3-init-xbg6x"
Jul 10 19:06:56.597881 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.597863    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-model-cache\") pod \"s3-init-xbg6x\" (UID: \"3f4ef870-3dc2-46e0-960a-ba52569f9ee9\") " pod="kserve/s3-init-xbg6x"
Jul 10 19:06:56.606701 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.606668    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt4c7\" (UniqueName: \"kubernetes.io/projected/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-kube-api-access-mt4c7\") pod \"s3-init-xbg6x\" (UID: \"3f4ef870-3dc2-46e0-960a-ba52569f9ee9\") " pod="kserve/s3-init-xbg6x"
Jul 10 19:06:56.678812 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.678745    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xbg6x"
Jul 10 19:06:56.804152 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:56.804128    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-xbg6x"]
Jul 10 19:06:56.806579 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:06:56.806544    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f4ef870_3dc2_46e0_960a_ba52569f9ee9.slice/crio-1d00b663623db1633fd1657df5801e3b6959828f25e5d9892bdb0a9bf9d711ea WatchSource:0}: Error finding container 1d00b663623db1633fd1657df5801e3b6959828f25e5d9892bdb0a9bf9d711ea: Status 404 returned error can't find the container with id 1d00b663623db1633fd1657df5801e3b6959828f25e5d9892bdb0a9bf9d711ea
Jul 10 19:06:57.538974 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:06:57.538865    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xbg6x" event={"ID":"3f4ef870-3dc2-46e0-960a-ba52569f9ee9","Type":"ContainerStarted","Data":"1d00b663623db1633fd1657df5801e3b6959828f25e5d9892bdb0a9bf9d711ea"}
Jul 10 19:07:01.559594 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:07:01.559556    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xbg6x" event={"ID":"3f4ef870-3dc2-46e0-960a-ba52569f9ee9","Type":"ContainerStarted","Data":"c9712244855bc5dcd63243a5a8bf8b22ccf94cce58c9388f637117ec3d948dcd"}
Jul 10 19:07:35.689688 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:07:35.689656    2567 generic.go:356] "Generic (PLEG): container finished" podID="3f4ef870-3dc2-46e0-960a-ba52569f9ee9" containerID="c9712244855bc5dcd63243a5a8bf8b22ccf94cce58c9388f637117ec3d948dcd" exitCode=0
Jul 10 19:07:35.690093 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:07:35.689703    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xbg6x" event={"ID":"3f4ef870-3dc2-46e0-960a-ba52569f9ee9","Type":"ContainerDied","Data":"c9712244855bc5dcd63243a5a8bf8b22ccf94cce58c9388f637117ec3d948dcd"}
Jul 10 19:07:40.273874 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:07:40.273848    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 10 19:07:40.711952 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:07:40.711903    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xbg6x" event={"ID":"3f4ef870-3dc2-46e0-960a-ba52569f9ee9","Type":"ContainerStarted","Data":"8a2cff6091b92e3ea271854e84c80683ec4f3115a3fd1513ef536bcf47447ae7"}
Jul 10 19:07:40.732599 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:07:40.732548    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-xbg6x" podStartSLOduration=2.196023823 podStartE2EDuration="44.732527279s" totalImagesPullingTime="8.804595454s" totalInitContainerRuntime="33.731908002s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:40.73232142 +0000 UTC m=+563.983968751" watchObservedRunningTime="2026-07-10 19:07:40.732527279 +0000 UTC m=+563.984174608"
Jul 10 19:08:25.640927 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:25.640878    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:08:26.956600 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:26.956567    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:08:45.959183 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:45.959147    2567 generic.go:356] "Generic (PLEG): container finished" podID="3f4ef870-3dc2-46e0-960a-ba52569f9ee9" containerID="8a2cff6091b92e3ea271854e84c80683ec4f3115a3fd1513ef536bcf47447ae7" exitCode=0
Jul 10 19:08:45.959551 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:45.959192    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xbg6x" event={"ID":"3f4ef870-3dc2-46e0-960a-ba52569f9ee9","Type":"ContainerDied","Data":"8a2cff6091b92e3ea271854e84c80683ec4f3115a3fd1513ef536bcf47447ae7"}
Jul 10 19:08:47.090845 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:47.090819    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xbg6x"
Jul 10 19:08:47.160207 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:47.160186    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-model-cache\") pod \"3f4ef870-3dc2-46e0-960a-ba52569f9ee9\" (UID: \"3f4ef870-3dc2-46e0-960a-ba52569f9ee9\") "
Jul 10 19:08:47.160323 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:47.160252    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt4c7\" (UniqueName: \"kubernetes.io/projected/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-kube-api-access-mt4c7\") pod \"3f4ef870-3dc2-46e0-960a-ba52569f9ee9\" (UID: \"3f4ef870-3dc2-46e0-960a-ba52569f9ee9\") "
Jul 10 19:08:47.162354 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:47.162326    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-kube-api-access-mt4c7" (OuterVolumeSpecName: "kube-api-access-mt4c7") pod "3f4ef870-3dc2-46e0-960a-ba52569f9ee9" (UID: "3f4ef870-3dc2-46e0-960a-ba52569f9ee9"). InnerVolumeSpecName "kube-api-access-mt4c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:08:47.179556 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:47.179524    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f4ef870-3dc2-46e0-960a-ba52569f9ee9" (UID: "3f4ef870-3dc2-46e0-960a-ba52569f9ee9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:08:47.261456 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:47.261406    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-model-cache\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:08:47.261456 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:47.261427    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt4c7\" (UniqueName: \"kubernetes.io/projected/3f4ef870-3dc2-46e0-960a-ba52569f9ee9-kube-api-access-mt4c7\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:08:47.968672 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:47.968645    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xbg6x"
Jul 10 19:08:47.968672 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:47.968644    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xbg6x" event={"ID":"3f4ef870-3dc2-46e0-960a-ba52569f9ee9","Type":"ContainerDied","Data":"1d00b663623db1633fd1657df5801e3b6959828f25e5d9892bdb0a9bf9d711ea"}
Jul 10 19:08:47.968860 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:47.968682    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d00b663623db1633fd1657df5801e3b6959828f25e5d9892bdb0a9bf9d711ea"
Jul 10 19:08:48.683726 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:08:48.683695    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:09:21.042690 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:21.042658    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:09:26.683862 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.683825    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-nqnp7"]
Jul 10 19:09:26.684371 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.684155    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4ef870-3dc2-46e0-960a-ba52569f9ee9" containerName="s3-init"
Jul 10 19:09:26.684371 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.684167    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4ef870-3dc2-46e0-960a-ba52569f9ee9" containerName="s3-init"
Jul 10 19:09:26.684371 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.684176    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4ef870-3dc2-46e0-960a-ba52569f9ee9" containerName="download-hf-model"
Jul 10 19:09:26.684371 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.684182    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4ef870-3dc2-46e0-960a-ba52569f9ee9" containerName="download-hf-model"
Jul 10 19:09:26.684371 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.684236    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f4ef870-3dc2-46e0-960a-ba52569f9ee9" containerName="s3-init"
Jul 10 19:09:26.686033 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.686017    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-nqnp7"
Jul 10 19:09:26.688385 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.688361    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 10 19:09:26.688385 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.688381    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-msbqc\""
Jul 10 19:09:26.694455 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.694434    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-nqnp7"]
Jul 10 19:09:26.763434 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.763408    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6832344-e91d-457f-abdc-e363b71c7f7f-model-cache\") pod \"s3-tls-init-custom-nqnp7\" (UID: \"c6832344-e91d-457f-abdc-e363b71c7f7f\") " pod="kserve/s3-tls-init-custom-nqnp7"
Jul 10 19:09:26.763543 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.763445    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxw9x\" (UniqueName: \"kubernetes.io/projected/c6832344-e91d-457f-abdc-e363b71c7f7f-kube-api-access-gxw9x\") pod \"s3-tls-init-custom-nqnp7\" (UID: \"c6832344-e91d-457f-abdc-e363b71c7f7f\") " pod="kserve/s3-tls-init-custom-nqnp7"
Jul 10 19:09:26.863775 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.863744    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6832344-e91d-457f-abdc-e363b71c7f7f-model-cache\") pod \"s3-tls-init-custom-nqnp7\" (UID: \"c6832344-e91d-457f-abdc-e363b71c7f7f\") " pod="kserve/s3-tls-init-custom-nqnp7"
Jul 10 19:09:26.863889 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.863790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxw9x\" (UniqueName: \"kubernetes.io/projected/c6832344-e91d-457f-abdc-e363b71c7f7f-kube-api-access-gxw9x\") pod \"s3-tls-init-custom-nqnp7\" (UID: \"c6832344-e91d-457f-abdc-e363b71c7f7f\") " pod="kserve/s3-tls-init-custom-nqnp7"
Jul 10 19:09:26.864119 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.864098    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6832344-e91d-457f-abdc-e363b71c7f7f-model-cache\") pod \"s3-tls-init-custom-nqnp7\" (UID: \"c6832344-e91d-457f-abdc-e363b71c7f7f\") " pod="kserve/s3-tls-init-custom-nqnp7"
Jul 10 19:09:26.871758 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:26.871738    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxw9x\" (UniqueName: \"kubernetes.io/projected/c6832344-e91d-457f-abdc-e363b71c7f7f-kube-api-access-gxw9x\") pod \"s3-tls-init-custom-nqnp7\" (UID: \"c6832344-e91d-457f-abdc-e363b71c7f7f\") " pod="kserve/s3-tls-init-custom-nqnp7"
Jul 10 19:09:27.004474 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:27.004397    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-nqnp7"
Jul 10 19:09:27.132813 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:27.132704    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-nqnp7"]
Jul 10 19:09:27.135394 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:09:27.135368    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6832344_e91d_457f_abdc_e363b71c7f7f.slice/crio-a0b4484efe0237bb30686cee8a6f6206f44931832b7d5e34426e738171ad5f6e WatchSource:0}: Error finding container a0b4484efe0237bb30686cee8a6f6206f44931832b7d5e34426e738171ad5f6e: Status 404 returned error can't find the container with id a0b4484efe0237bb30686cee8a6f6206f44931832b7d5e34426e738171ad5f6e
Jul 10 19:09:28.118184 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:28.118146    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nqnp7" event={"ID":"c6832344-e91d-457f-abdc-e363b71c7f7f","Type":"ContainerStarted","Data":"49693cbf4a68927d08027497bf3ca3e7564eadcf04dd73c729ca9c6208b05d43"}
Jul 10 19:09:28.118672 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:28.118194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nqnp7" event={"ID":"c6832344-e91d-457f-abdc-e363b71c7f7f","Type":"ContainerStarted","Data":"a0b4484efe0237bb30686cee8a6f6206f44931832b7d5e34426e738171ad5f6e"}
Jul 10 19:09:30.127903 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:30.127867    2567 generic.go:356] "Generic (PLEG): container finished" podID="c6832344-e91d-457f-abdc-e363b71c7f7f" containerID="49693cbf4a68927d08027497bf3ca3e7564eadcf04dd73c729ca9c6208b05d43" exitCode=0
Jul 10 19:09:30.128349 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:30.127937    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nqnp7" event={"ID":"c6832344-e91d-457f-abdc-e363b71c7f7f","Type":"ContainerDied","Data":"49693cbf4a68927d08027497bf3ca3e7564eadcf04dd73c729ca9c6208b05d43"}
Jul 10 19:09:31.133446 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:31.133416    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nqnp7" event={"ID":"c6832344-e91d-457f-abdc-e363b71c7f7f","Type":"ContainerStarted","Data":"fa6bd1252958c7615f2fd58e2ce84d78dcb07ea0fe958c3217a9a50c48276d1f"}
Jul 10 19:09:31.151381 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:31.151328    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-nqnp7" podStartSLOduration=2.922776494 podStartE2EDuration="5.151310588s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.228534094s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:31.149668903 +0000 UTC m=+674.401316230" watchObservedRunningTime="2026-07-10 19:09:31.151310588 +0000 UTC m=+674.402957950"
Jul 10 19:09:36.154469 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:36.154435    2567 generic.go:356] "Generic (PLEG): container finished" podID="c6832344-e91d-457f-abdc-e363b71c7f7f" containerID="fa6bd1252958c7615f2fd58e2ce84d78dcb07ea0fe958c3217a9a50c48276d1f" exitCode=0
Jul 10 19:09:36.154859 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:36.154507    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nqnp7" event={"ID":"c6832344-e91d-457f-abdc-e363b71c7f7f","Type":"ContainerDied","Data":"fa6bd1252958c7615f2fd58e2ce84d78dcb07ea0fe958c3217a9a50c48276d1f"}
Jul 10 19:09:37.295354 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:37.295331    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-nqnp7"
Jul 10 19:09:37.351632 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:37.351606    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxw9x\" (UniqueName: \"kubernetes.io/projected/c6832344-e91d-457f-abdc-e363b71c7f7f-kube-api-access-gxw9x\") pod \"c6832344-e91d-457f-abdc-e363b71c7f7f\" (UID: \"c6832344-e91d-457f-abdc-e363b71c7f7f\") "
Jul 10 19:09:37.351738 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:37.351657    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6832344-e91d-457f-abdc-e363b71c7f7f-model-cache\") pod \"c6832344-e91d-457f-abdc-e363b71c7f7f\" (UID: \"c6832344-e91d-457f-abdc-e363b71c7f7f\") "
Jul 10 19:09:37.353620 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:37.353589    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6832344-e91d-457f-abdc-e363b71c7f7f-kube-api-access-gxw9x" (OuterVolumeSpecName: "kube-api-access-gxw9x") pod "c6832344-e91d-457f-abdc-e363b71c7f7f" (UID: "c6832344-e91d-457f-abdc-e363b71c7f7f"). InnerVolumeSpecName "kube-api-access-gxw9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:09:37.374428 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:37.374397    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6832344-e91d-457f-abdc-e363b71c7f7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6832344-e91d-457f-abdc-e363b71c7f7f" (UID: "c6832344-e91d-457f-abdc-e363b71c7f7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:09:37.452470 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:37.452442    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxw9x\" (UniqueName: \"kubernetes.io/projected/c6832344-e91d-457f-abdc-e363b71c7f7f-kube-api-access-gxw9x\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:09:37.452470 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:37.452465    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6832344-e91d-457f-abdc-e363b71c7f7f-model-cache\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:09:38.164654 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:38.164629    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-nqnp7"
Jul 10 19:09:38.164814 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:38.164626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nqnp7" event={"ID":"c6832344-e91d-457f-abdc-e363b71c7f7f","Type":"ContainerDied","Data":"a0b4484efe0237bb30686cee8a6f6206f44931832b7d5e34426e738171ad5f6e"}
Jul 10 19:09:38.164814 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:38.164739    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0b4484efe0237bb30686cee8a6f6206f44931832b7d5e34426e738171ad5f6e"
Jul 10 19:09:38.758670 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:38.758644    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:09:39.034266 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:39.034188    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:09:40.967370 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:40.967338    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-qjdxk"]
Jul 10 19:09:40.967823 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:40.967653    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6832344-e91d-457f-abdc-e363b71c7f7f" containerName="s3-tls-init-custom"
Jul 10 19:09:40.967823 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:40.967664    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6832344-e91d-457f-abdc-e363b71c7f7f" containerName="s3-tls-init-custom"
Jul 10 19:09:40.967823 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:40.967680    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6832344-e91d-457f-abdc-e363b71c7f7f" containerName="download-hf-model"
Jul 10 19:09:40.967823 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:40.967686    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6832344-e91d-457f-abdc-e363b71c7f7f" containerName="download-hf-model"
Jul 10 19:09:40.967823 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:40.967735    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6832344-e91d-457f-abdc-e363b71c7f7f" containerName="s3-tls-init-custom"
Jul 10 19:09:40.970986 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:40.970970    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-qjdxk"
Jul 10 19:09:40.973264 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:40.973242    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 10 19:09:40.973380 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:40.973242    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-msbqc\""
Jul 10 19:09:40.978024 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:40.978001    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-qjdxk"]
Jul 10 19:09:41.075692 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:41.075652    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7jx\" (UniqueName: \"kubernetes.io/projected/fc4c6cad-414d-4215-aabe-44a039cec360-kube-api-access-th7jx\") pod \"s3-tls-init-serving-qjdxk\" (UID: \"fc4c6cad-414d-4215-aabe-44a039cec360\") " pod="kserve/s3-tls-init-serving-qjdxk"
Jul 10 19:09:41.075824 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:41.075719    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc4c6cad-414d-4215-aabe-44a039cec360-model-cache\") pod \"s3-tls-init-serving-qjdxk\" (UID: \"fc4c6cad-414d-4215-aabe-44a039cec360\") " pod="kserve/s3-tls-init-serving-qjdxk"
Jul 10 19:09:41.176979 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:41.176952    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7jx\" (UniqueName: \"kubernetes.io/projected/fc4c6cad-414d-4215-aabe-44a039cec360-kube-api-access-th7jx\") pod \"s3-tls-init-serving-qjdxk\" (UID: \"fc4c6cad-414d-4215-aabe-44a039cec360\") " pod="kserve/s3-tls-init-serving-qjdxk"
Jul 10 19:09:41.177101 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:41.176989    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc4c6cad-414d-4215-aabe-44a039cec360-model-cache\") pod \"s3-tls-init-serving-qjdxk\" (UID: \"fc4c6cad-414d-4215-aabe-44a039cec360\") " pod="kserve/s3-tls-init-serving-qjdxk"
Jul 10 19:09:41.177305 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:41.177290    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc4c6cad-414d-4215-aabe-44a039cec360-model-cache\") pod \"s3-tls-init-serving-qjdxk\" (UID: \"fc4c6cad-414d-4215-aabe-44a039cec360\") " pod="kserve/s3-tls-init-serving-qjdxk"
Jul 10 19:09:41.185345 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:41.185321    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7jx\" (UniqueName: \"kubernetes.io/projected/fc4c6cad-414d-4215-aabe-44a039cec360-kube-api-access-th7jx\") pod \"s3-tls-init-serving-qjdxk\" (UID: \"fc4c6cad-414d-4215-aabe-44a039cec360\") " pod="kserve/s3-tls-init-serving-qjdxk"
Jul 10 19:09:41.286738 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:41.286681    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-qjdxk"
Jul 10 19:09:41.412425 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:41.412403    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-qjdxk"]
Jul 10 19:09:41.415242 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:09:41.415217    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc4c6cad_414d_4215_aabe_44a039cec360.slice/crio-29ee884e8cabce1706370fdcdf63a2ba87872a8c67c207bb58bdf196d874f155 WatchSource:0}: Error finding container 29ee884e8cabce1706370fdcdf63a2ba87872a8c67c207bb58bdf196d874f155: Status 404 returned error can't find the container with id 29ee884e8cabce1706370fdcdf63a2ba87872a8c67c207bb58bdf196d874f155
Jul 10 19:09:42.180148 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:42.180111    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qjdxk" event={"ID":"fc4c6cad-414d-4215-aabe-44a039cec360","Type":"ContainerStarted","Data":"0272b5dbb695ba3a357cf4f9a4ca79ef3d8e024deec8e1c855d0cdc9bb132bc8"}
Jul 10 19:09:42.180148 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:42.180152    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qjdxk" event={"ID":"fc4c6cad-414d-4215-aabe-44a039cec360","Type":"ContainerStarted","Data":"29ee884e8cabce1706370fdcdf63a2ba87872a8c67c207bb58bdf196d874f155"}
Jul 10 19:09:44.189517 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:44.189485    2567 generic.go:356] "Generic (PLEG): container finished" podID="fc4c6cad-414d-4215-aabe-44a039cec360" containerID="0272b5dbb695ba3a357cf4f9a4ca79ef3d8e024deec8e1c855d0cdc9bb132bc8" exitCode=0
Jul 10 19:09:44.189895 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:44.189563    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qjdxk" event={"ID":"fc4c6cad-414d-4215-aabe-44a039cec360","Type":"ContainerDied","Data":"0272b5dbb695ba3a357cf4f9a4ca79ef3d8e024deec8e1c855d0cdc9bb132bc8"}
Jul 10 19:09:45.195388 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:45.195352    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qjdxk" event={"ID":"fc4c6cad-414d-4215-aabe-44a039cec360","Type":"ContainerStarted","Data":"119a61d17799037d367c1332a2b4107ffab73cbd84f10a2d33802c1fc619e584"}
Jul 10 19:09:45.212926 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:45.212859    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-qjdxk" podStartSLOduration=3.051684911 podStartE2EDuration="5.212843395s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.161158484s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:45.211706115 +0000 UTC m=+688.463353446" watchObservedRunningTime="2026-07-10 19:09:45.212843395 +0000 UTC m=+688.464490723"
Jul 10 19:09:52.225664 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:52.225626    2567 generic.go:356] "Generic (PLEG): container finished" podID="fc4c6cad-414d-4215-aabe-44a039cec360" containerID="119a61d17799037d367c1332a2b4107ffab73cbd84f10a2d33802c1fc619e584" exitCode=0
Jul 10 19:09:52.226119 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:52.225698    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qjdxk" event={"ID":"fc4c6cad-414d-4215-aabe-44a039cec360","Type":"ContainerDied","Data":"119a61d17799037d367c1332a2b4107ffab73cbd84f10a2d33802c1fc619e584"}
Jul 10 19:09:53.374508 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:53.374484    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-qjdxk"
Jul 10 19:09:53.478776 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:53.478747    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc4c6cad-414d-4215-aabe-44a039cec360-model-cache\") pod \"fc4c6cad-414d-4215-aabe-44a039cec360\" (UID: \"fc4c6cad-414d-4215-aabe-44a039cec360\") "
Jul 10 19:09:53.478937 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:53.478851    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th7jx\" (UniqueName: \"kubernetes.io/projected/fc4c6cad-414d-4215-aabe-44a039cec360-kube-api-access-th7jx\") pod \"fc4c6cad-414d-4215-aabe-44a039cec360\" (UID: \"fc4c6cad-414d-4215-aabe-44a039cec360\") "
Jul 10 19:09:53.480839 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:53.480784    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4c6cad-414d-4215-aabe-44a039cec360-kube-api-access-th7jx" (OuterVolumeSpecName: "kube-api-access-th7jx") pod "fc4c6cad-414d-4215-aabe-44a039cec360" (UID: "fc4c6cad-414d-4215-aabe-44a039cec360"). InnerVolumeSpecName "kube-api-access-th7jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:09:53.511165 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:53.511133    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4c6cad-414d-4215-aabe-44a039cec360-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc4c6cad-414d-4215-aabe-44a039cec360" (UID: "fc4c6cad-414d-4215-aabe-44a039cec360"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:09:53.579938 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:53.579898    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7jx\" (UniqueName: \"kubernetes.io/projected/fc4c6cad-414d-4215-aabe-44a039cec360-kube-api-access-th7jx\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:09:53.579938 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:53.579937    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc4c6cad-414d-4215-aabe-44a039cec360-model-cache\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:09:54.236140 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:54.236098    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qjdxk" event={"ID":"fc4c6cad-414d-4215-aabe-44a039cec360","Type":"ContainerDied","Data":"29ee884e8cabce1706370fdcdf63a2ba87872a8c67c207bb58bdf196d874f155"}
Jul 10 19:09:54.236140 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:54.236144    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ee884e8cabce1706370fdcdf63a2ba87872a8c67c207bb58bdf196d874f155"
Jul 10 19:09:54.236399 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:54.236143    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-qjdxk"
Jul 10 19:09:54.587876 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:54.587848    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:09:54.687583 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:09:54.687547    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:00.816872 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:00.816823    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:05.703835 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.703800    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:05.783359 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.783329    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"]
Jul 10 19:10:05.783650 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.783636    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4c6cad-414d-4215-aabe-44a039cec360" containerName="download-hf-model"
Jul 10 19:10:05.783650 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.783650    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4c6cad-414d-4215-aabe-44a039cec360" containerName="download-hf-model"
Jul 10 19:10:05.783816 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.783668    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4c6cad-414d-4215-aabe-44a039cec360" containerName="s3-tls-init-serving"
Jul 10 19:10:05.783816 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.783678    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4c6cad-414d-4215-aabe-44a039cec360" containerName="s3-tls-init-serving"
Jul 10 19:10:05.783816 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.783729    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4c6cad-414d-4215-aabe-44a039cec360" containerName="s3-tls-init-serving"
Jul 10 19:10:05.787481 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.787464    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.790023 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.790001    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 10 19:10:05.790163 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.790038    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 10 19:10:05.790163 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.790054    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 19:10:05.790163 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.790083    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:10:05.790342 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.790183    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z7zjv\""
Jul 10 19:10:05.790425 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.790411    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 10 19:10:05.791003 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.790985    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 19:10:05.801712 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.801694    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"]
Jul 10 19:10:05.876671 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.876638    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fcb7138-efd4-4092-b246-8f1726f843c7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.876671 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.876675    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g584n\" (UniqueName: \"kubernetes.io/projected/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-api-access-g584n\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.876878 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.876703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.876878 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.876785    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fcb7138-efd4-4092-b246-8f1726f843c7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.876878 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.876862    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.879975 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.879949    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:05.978213 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.978129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.978213 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.978172    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fcb7138-efd4-4092-b246-8f1726f843c7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.978213 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.978200    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g584n\" (UniqueName: \"kubernetes.io/projected/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-api-access-g584n\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.978445 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.978236    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.978445 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.978272    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fcb7138-efd4-4092-b246-8f1726f843c7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.978692 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.978669    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fcb7138-efd4-4092-b246-8f1726f843c7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.978994 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.978973    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.979053 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.979010    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.980616 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.980600    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fcb7138-efd4-4092-b246-8f1726f843c7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:05.986230 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:05.986211    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g584n\" (UniqueName: \"kubernetes.io/projected/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-api-access-g584n\") pod \"isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:06.099019 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:06.098981    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:06.231079 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:06.231054    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"]
Jul 10 19:10:06.233351 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:10:06.233326    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcb7138_efd4_4092_b246_8f1726f843c7.slice/crio-e876943698272b3becc24fb5d2a1fde065f30407d878f80d086813f318762eba WatchSource:0}: Error finding container e876943698272b3becc24fb5d2a1fde065f30407d878f80d086813f318762eba: Status 404 returned error can't find the container with id e876943698272b3becc24fb5d2a1fde065f30407d878f80d086813f318762eba
Jul 10 19:10:06.285184 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:06.285158    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" event={"ID":"6fcb7138-efd4-4092-b246-8f1726f843c7","Type":"ContainerStarted","Data":"e876943698272b3becc24fb5d2a1fde065f30407d878f80d086813f318762eba"}
Jul 10 19:10:07.290984 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:07.290944    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" event={"ID":"6fcb7138-efd4-4092-b246-8f1726f843c7","Type":"ContainerStarted","Data":"d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a"}
Jul 10 19:10:10.303661 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:10.303574    2567 generic.go:356] "Generic (PLEG): container finished" podID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerID="d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a" exitCode=0
Jul 10 19:10:10.303661 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:10.303645    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" event={"ID":"6fcb7138-efd4-4092-b246-8f1726f843c7","Type":"ContainerDied","Data":"d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a"}
Jul 10 19:10:25.381933 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:25.381878    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" event={"ID":"6fcb7138-efd4-4092-b246-8f1726f843c7","Type":"ContainerStarted","Data":"404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5"}
Jul 10 19:10:28.398071 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:28.398033    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" event={"ID":"6fcb7138-efd4-4092-b246-8f1726f843c7","Type":"ContainerStarted","Data":"c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596"}
Jul 10 19:10:30.409380 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:30.409339    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" event={"ID":"6fcb7138-efd4-4092-b246-8f1726f843c7","Type":"ContainerStarted","Data":"909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6"}
Jul 10 19:10:30.409964 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:30.409939    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:30.410081 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:30.409992    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:30.410948 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:30.410901    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:10:30.430964 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:30.430898    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podStartSLOduration=2.669477628 podStartE2EDuration="25.430883986s" totalImagesPullingTime="19.000175305s" totalInitContainerRuntime="3.761231053s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:30.429615765 +0000 UTC m=+733.681263105" watchObservedRunningTime="2026-07-10 19:10:30.430883986 +0000 UTC m=+733.682531314"
Jul 10 19:10:31.414021 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:31.413978    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:10:31.415226 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:31.415181    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:31.419920 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:31.419887    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:32.418054 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:32.418013    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:10:32.418528 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:32.418496    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:32.421867 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:32.421847    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:10:33.421414 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:33.421373    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:10:33.421785 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:33.421678    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:43.422207 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:43.422163    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:10:43.422701 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:43.422545    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:53.421716 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:53.421673    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:10:53.422138 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:10:53.422107    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:03.421556 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:03.421510    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:11:03.421994 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:03.421955    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:13.421660 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:13.421616    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:11:13.422125 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:13.422097    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:23.422055 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:23.422011    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:11:23.422524 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:23.422348    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:33.422176 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:33.422135    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:11:33.422657 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:33.422511    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:43.422476 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:43.422397    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:11:43.422988 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:43.422965    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:11:50.881652 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:50.881622    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"]
Jul 10 19:11:50.882304 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:50.881963    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" containerID="cri-o://404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5" gracePeriod=30
Jul 10 19:11:50.882304 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:50.881980    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" containerID="cri-o://909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6" gracePeriod=30
Jul 10 19:11:50.882304 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:50.882058    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kube-rbac-proxy" containerID="cri-o://c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596" gracePeriod=30
Jul 10 19:11:50.937586 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:50.937557    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:51.000208 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.000182    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"]
Jul 10 19:11:51.001595 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.001566    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:51.002859 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.002841    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.005324 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.005301    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 10 19:11:51.005414 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.005308    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 10 19:11:51.015214 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.015185    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"]
Jul 10 19:11:51.019641 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.019620    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzfg8\" (UniqueName: \"kubernetes.io/projected/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-api-access-zzfg8\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.019738 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.019649    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.019738 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.019672    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61245d75-9889-4b3f-ba6c-4b671b6607c9-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.019823 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.019737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61245d75-9889-4b3f-ba6c-4b671b6607c9-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.019823 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.019785    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.120798 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.120768    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61245d75-9889-4b3f-ba6c-4b671b6607c9-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.120977 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.120822    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.120977 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.120848    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzfg8\" (UniqueName: \"kubernetes.io/projected/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-api-access-zzfg8\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.120977 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.120868    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.120977 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.120894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61245d75-9889-4b3f-ba6c-4b671b6607c9-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.121203 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:11:51.121034    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 10 19:11:51.121203 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:11:51.121095    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61245d75-9889-4b3f-ba6c-4b671b6607c9-proxy-tls podName:61245d75-9889-4b3f-ba6c-4b671b6607c9 nodeName:}" failed. No retries permitted until 2026-07-10 19:11:51.621076322 +0000 UTC m=+814.872723642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/61245d75-9889-4b3f-ba6c-4b671b6607c9-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" (UID: "61245d75-9889-4b3f-ba6c-4b671b6607c9") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 10 19:11:51.121320 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.121278    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61245d75-9889-4b3f-ba6c-4b671b6607c9-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.121613 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.121591    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.121688 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.121593    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.129206 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.129174    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzfg8\" (UniqueName: \"kubernetes.io/projected/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-api-access-zzfg8\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.302354 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.302327    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:51.626596 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.626505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61245d75-9889-4b3f-ba6c-4b671b6607c9-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.628862 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.628830    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61245d75-9889-4b3f-ba6c-4b671b6607c9-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:51.725419 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.725383    2567 generic.go:356] "Generic (PLEG): container finished" podID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerID="c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596" exitCode=2
Jul 10 19:11:51.725572 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.725467    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" event={"ID":"6fcb7138-efd4-4092-b246-8f1726f843c7","Type":"ContainerDied","Data":"c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596"}
Jul 10 19:11:51.912842 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:51.912756    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:52.059013 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:52.058985    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"]
Jul 10 19:11:52.061869 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:11:52.061830    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61245d75_9889_4b3f_ba6c_4b671b6607c9.slice/crio-bb6edcb5280af919b01fcb9bf7f32bf54f71b47185a3464c47c9d2adbb00ae82 WatchSource:0}: Error finding container bb6edcb5280af919b01fcb9bf7f32bf54f71b47185a3464c47c9d2adbb00ae82: Status 404 returned error can't find the container with id bb6edcb5280af919b01fcb9bf7f32bf54f71b47185a3464c47c9d2adbb00ae82
Jul 10 19:11:52.418393 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:52.418356    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 10 19:11:52.730593 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:52.730502    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" event={"ID":"61245d75-9889-4b3f-ba6c-4b671b6607c9","Type":"ContainerStarted","Data":"f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d"}
Jul 10 19:11:52.730593 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:52.730583    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" event={"ID":"61245d75-9889-4b3f-ba6c-4b671b6607c9","Type":"ContainerStarted","Data":"bb6edcb5280af919b01fcb9bf7f32bf54f71b47185a3464c47c9d2adbb00ae82"}
Jul 10 19:11:53.422425 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:53.422388    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:53.422879 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:53.422852    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:11:55.745307 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:55.745275    2567 generic.go:356] "Generic (PLEG): container finished" podID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerID="404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5" exitCode=0
Jul 10 19:11:55.745641 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:55.745325    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" event={"ID":"6fcb7138-efd4-4092-b246-8f1726f843c7","Type":"ContainerDied","Data":"404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5"}
Jul 10 19:11:56.750495 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:56.750462    2567 generic.go:356] "Generic (PLEG): container finished" podID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerID="f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d" exitCode=0
Jul 10 19:11:56.750868 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:56.750533    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" event={"ID":"61245d75-9889-4b3f-ba6c-4b671b6607c9","Type":"ContainerDied","Data":"f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d"}
Jul 10 19:11:57.418312 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:57.418273    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 10 19:11:57.756697 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:57.756614    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" event={"ID":"61245d75-9889-4b3f-ba6c-4b671b6607c9","Type":"ContainerStarted","Data":"99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8"}
Jul 10 19:11:57.756697 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:57.756685    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" event={"ID":"61245d75-9889-4b3f-ba6c-4b671b6607c9","Type":"ContainerStarted","Data":"fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596"}
Jul 10 19:11:57.757165 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:57.756709    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" event={"ID":"61245d75-9889-4b3f-ba6c-4b671b6607c9","Type":"ContainerStarted","Data":"43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd"}
Jul 10 19:11:57.757165 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:57.756931    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:58.761144 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:58.761108    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:58.761601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:58.761179    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:11:58.762705 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:58.762674    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:11:58.763386 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:58.763361    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:59.765076 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:59.765027    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:11:59.765528 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:11:59.765372    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:02.418873 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:02.418826    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 10 19:12:02.419307 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:02.419004    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:12:02.442669 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:02.442603    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podStartSLOduration=8.523911173 podStartE2EDuration="12.442581648s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.918670475s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:11:57.781120242 +0000 UTC m=+821.032767586" watchObservedRunningTime="2026-07-10 19:12:02.442581648 +0000 UTC m=+825.694228977"
Jul 10 19:12:03.422200 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:03.422158    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:03.423410 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:03.423384    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:12:04.769632 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:04.769598    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:12:04.770286 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:04.770249    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:12:04.770489 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:04.770460    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:07.418701 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:07.418662    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 10 19:12:12.418220 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:12.418177    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 10 19:12:13.422001 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:13.421963    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:13.422379 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:13.422083    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:12:13.423208 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:13.423182    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:12:13.423313 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:13.423283    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:12:14.770031 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:14.769989    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:12:14.770543 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:14.770519    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:17.418535 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:17.418486    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 10 19:12:21.031751 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.031724    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:12:21.175047 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.174970    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fcb7138-efd4-4092-b246-8f1726f843c7-proxy-tls\") pod \"6fcb7138-efd4-4092-b246-8f1726f843c7\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") "
Jul 10 19:12:21.175047 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.175006    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-cabundle-cert\") pod \"6fcb7138-efd4-4092-b246-8f1726f843c7\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") "
Jul 10 19:12:21.175267 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.175057    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g584n\" (UniqueName: \"kubernetes.io/projected/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-api-access-g584n\") pod \"6fcb7138-efd4-4092-b246-8f1726f843c7\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") "
Jul 10 19:12:21.175267 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.175076    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-rbac-proxy-sar-config\") pod \"6fcb7138-efd4-4092-b246-8f1726f843c7\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") "
Jul 10 19:12:21.175267 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.175138    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fcb7138-efd4-4092-b246-8f1726f843c7-kserve-provision-location\") pod \"6fcb7138-efd4-4092-b246-8f1726f843c7\" (UID: \"6fcb7138-efd4-4092-b246-8f1726f843c7\") "
Jul 10 19:12:21.175435 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.175411    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6fcb7138-efd4-4092-b246-8f1726f843c7" (UID: "6fcb7138-efd4-4092-b246-8f1726f843c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:12:21.175484 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.175460    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6fcb7138-efd4-4092-b246-8f1726f843c7" (UID: "6fcb7138-efd4-4092-b246-8f1726f843c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:12:21.175520 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.175483    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcb7138-efd4-4092-b246-8f1726f843c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6fcb7138-efd4-4092-b246-8f1726f843c7" (UID: "6fcb7138-efd4-4092-b246-8f1726f843c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:12:21.177190 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.177167    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-api-access-g584n" (OuterVolumeSpecName: "kube-api-access-g584n") pod "6fcb7138-efd4-4092-b246-8f1726f843c7" (UID: "6fcb7138-efd4-4092-b246-8f1726f843c7"). InnerVolumeSpecName "kube-api-access-g584n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:12:21.177320 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.177220    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fcb7138-efd4-4092-b246-8f1726f843c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6fcb7138-efd4-4092-b246-8f1726f843c7" (UID: "6fcb7138-efd4-4092-b246-8f1726f843c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:12:21.275723 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.275694    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fcb7138-efd4-4092-b246-8f1726f843c7-kserve-provision-location\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:12:21.275723 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.275720    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fcb7138-efd4-4092-b246-8f1726f843c7-proxy-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:12:21.275867 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.275731    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-cabundle-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:12:21.275867 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.275739    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g584n\" (UniqueName: \"kubernetes.io/projected/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-api-access-g584n\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:12:21.275867 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.275747    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fcb7138-efd4-4092-b246-8f1726f843c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:12:21.865076 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.865041    2567 generic.go:356] "Generic (PLEG): container finished" podID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerID="909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6" exitCode=0
Jul 10 19:12:21.865253 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.865083    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" event={"ID":"6fcb7138-efd4-4092-b246-8f1726f843c7","Type":"ContainerDied","Data":"909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6"}
Jul 10 19:12:21.865253 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.865113    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"
Jul 10 19:12:21.865253 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.865138    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l" event={"ID":"6fcb7138-efd4-4092-b246-8f1726f843c7","Type":"ContainerDied","Data":"e876943698272b3becc24fb5d2a1fde065f30407d878f80d086813f318762eba"}
Jul 10 19:12:21.865253 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.865170    2567 scope.go:117] "RemoveContainer" containerID="909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6"
Jul 10 19:12:21.873309 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.873293    2567 scope.go:117] "RemoveContainer" containerID="c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596"
Jul 10 19:12:21.880651 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.880634    2567 scope.go:117] "RemoveContainer" containerID="404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5"
Jul 10 19:12:21.885696 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.885670    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"]
Jul 10 19:12:21.887771 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.887751    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-d68dc49bd-n2f9l"]
Jul 10 19:12:21.889658 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.889532    2567 scope.go:117] "RemoveContainer" containerID="d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a"
Jul 10 19:12:21.896540 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.896522    2567 scope.go:117] "RemoveContainer" containerID="909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6"
Jul 10 19:12:21.896823 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:12:21.896805    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6\": container with ID starting with 909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6 not found: ID does not exist" containerID="909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6"
Jul 10 19:12:21.896882 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.896831    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6"} err="failed to get container status \"909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6\": rpc error: code = NotFound desc = could not find container \"909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6\": container with ID starting with 909b1d9c7a35a399e3992c1df2bd99df21118ed14a06a7e16628efb2c4848fd6 not found: ID does not exist"
Jul 10 19:12:21.896882 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.896847    2567 scope.go:117] "RemoveContainer" containerID="c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596"
Jul 10 19:12:21.897090 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:12:21.897073    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596\": container with ID starting with c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596 not found: ID does not exist" containerID="c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596"
Jul 10 19:12:21.897130 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.897097    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596"} err="failed to get container status \"c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596\": rpc error: code = NotFound desc = could not find container \"c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596\": container with ID starting with c2d9376a1e94559c46942cf84d0a1a46e60f0ab929d4ce4e6cbaed3656735596 not found: ID does not exist"
Jul 10 19:12:21.897130 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.897112    2567 scope.go:117] "RemoveContainer" containerID="404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5"
Jul 10 19:12:21.897326 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:12:21.897312    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5\": container with ID starting with 404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5 not found: ID does not exist" containerID="404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5"
Jul 10 19:12:21.897362 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.897330    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5"} err="failed to get container status \"404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5\": rpc error: code = NotFound desc = could not find container \"404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5\": container with ID starting with 404bdc1c99729a460ed903af629d9619e57b12494d0a2d324c90b6a4a5ccefd5 not found: ID does not exist"
Jul 10 19:12:21.897362 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.897343    2567 scope.go:117] "RemoveContainer" containerID="d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a"
Jul 10 19:12:21.897535 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:12:21.897521    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a\": container with ID starting with d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a not found: ID does not exist" containerID="d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a"
Jul 10 19:12:21.897579 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:21.897538    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a"} err="failed to get container status \"d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a\": rpc error: code = NotFound desc = could not find container \"d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a\": container with ID starting with d7102a58eceff1779a526536c2d799362a3f01d95050cc3fd5a3f501da5fdd5a not found: ID does not exist"
Jul 10 19:12:23.421158 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:23.421125    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" path="/var/lib/kubelet/pods/6fcb7138-efd4-4092-b246-8f1726f843c7/volumes"
Jul 10 19:12:24.770397 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:24.770355    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:12:24.770869 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:24.770845    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:34.770780 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:34.770739    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:12:34.771287 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:34.771261    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:44.770992 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:44.770947    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:12:44.771460 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:44.771435    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:54.770151 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:54.770103    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:12:54.770554 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:12:54.770498    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:04.770404 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:04.770361    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:13:04.770797 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:04.770771    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:14.414134 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:14.414045    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:13:14.414628 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:14.414300    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:13:26.096712 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.096678    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"]
Jul 10 19:13:26.098259 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.097050    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" containerID="cri-o://43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd" gracePeriod=30
Jul 10 19:13:26.098259 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.097142    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kube-rbac-proxy" containerID="cri-o://fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596" gracePeriod=30
Jul 10 19:13:26.098259 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.097142    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" containerID="cri-o://99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8" gracePeriod=30
Jul 10 19:13:26.149187 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.149161    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:13:26.189607 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.189578    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"]
Jul 10 19:13:26.189887 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.189875    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container"
Jul 10 19:13:26.189964 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.189888    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container"
Jul 10 19:13:26.189964 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.189897    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="storage-initializer"
Jul 10 19:13:26.189964 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.189918    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="storage-initializer"
Jul 10 19:13:26.189964 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.189935    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kube-rbac-proxy"
Jul 10 19:13:26.189964 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.189943    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kube-rbac-proxy"
Jul 10 19:13:26.189964 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.189964    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent"
Jul 10 19:13:26.190165 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.189969    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent"
Jul 10 19:13:26.190165 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.190015    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="agent"
Jul 10 19:13:26.190165 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.190023    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kube-rbac-proxy"
Jul 10 19:13:26.190165 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.190030    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fcb7138-efd4-4092-b246-8f1726f843c7" containerName="kserve-container"
Jul 10 19:13:26.192158 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.192142    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:26.196395 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.196375    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 10 19:13:26.196395 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.196384    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 10 19:13:26.209006 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.208975    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"]
Jul 10 19:13:26.227985 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.227967    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:13:26.256344 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.256318    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2td7\" (UniqueName: \"kubernetes.io/projected/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-api-access-j2td7\") pod \"message-dumper-predictor-6575bcfdfd-828h5\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:26.256471 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.256405    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-828h5\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:26.256471 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.256453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-828h5\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:26.357517 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.357461    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-828h5\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:26.357517 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.357501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-828h5\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:26.357652 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.357562    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2td7\" (UniqueName: \"kubernetes.io/projected/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-api-access-j2td7\") pod \"message-dumper-predictor-6575bcfdfd-828h5\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:26.357727 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:13:26.357705    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 10 19:13:26.357791 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:13:26.357779    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-proxy-tls podName:d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f nodeName:}" failed. No retries permitted until 2026-07-10 19:13:26.857759064 +0000 UTC m=+910.109406376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-828h5" (UID: "d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f") : secret "message-dumper-predictor-serving-cert" not found
Jul 10 19:13:26.358164 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.358148    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-828h5\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:26.366589 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.366566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2td7\" (UniqueName: \"kubernetes.io/projected/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-api-access-j2td7\") pod \"message-dumper-predictor-6575bcfdfd-828h5\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:26.506495 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.506452    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:13:26.861111 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.861076    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-828h5\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:26.863383 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:26.863353    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-828h5\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:27.102041 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:27.102007    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:27.137633 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:27.137600    2567 generic.go:356] "Generic (PLEG): container finished" podID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerID="fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596" exitCode=2
Jul 10 19:13:27.138015 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:27.137667    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" event={"ID":"61245d75-9889-4b3f-ba6c-4b671b6607c9","Type":"ContainerDied","Data":"fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596"}
Jul 10 19:13:27.233553 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:27.233522    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"]
Jul 10 19:13:27.235902 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:13:27.235878    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bd1df1_ea5a_489a_aaf1_cdd6ac3b295f.slice/crio-eb85df886a0ed3b6ca173adac2e870b9f613aef87d742b945c44b5cdd23d77ec WatchSource:0}: Error finding container eb85df886a0ed3b6ca173adac2e870b9f613aef87d742b945c44b5cdd23d77ec: Status 404 returned error can't find the container with id eb85df886a0ed3b6ca173adac2e870b9f613aef87d742b945c44b5cdd23d77ec
Jul 10 19:13:27.237802 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:27.237785    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:13:28.143345 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:28.143296    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5" event={"ID":"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f","Type":"ContainerStarted","Data":"eb85df886a0ed3b6ca173adac2e870b9f613aef87d742b945c44b5cdd23d77ec"}
Jul 10 19:13:29.148793 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:29.148754    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5" event={"ID":"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f","Type":"ContainerStarted","Data":"de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8"}
Jul 10 19:13:29.149202 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:29.148814    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5" event={"ID":"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f","Type":"ContainerStarted","Data":"061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d"}
Jul 10 19:13:29.149202 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:29.148918    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:29.169269 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:29.169218    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5" podStartSLOduration=1.922947889 podStartE2EDuration="3.169204636s" totalImagesPullingTime="1.246256747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:13:29.169192129 +0000 UTC m=+912.420839458" watchObservedRunningTime="2026-07-10 19:13:29.169204636 +0000 UTC m=+912.420851964"
Jul 10 19:13:29.765897 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:29.765847    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 10 19:13:30.152582 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:30.152551    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:30.154061 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:30.154037    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:31.159099 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:31.159065    2567 generic.go:356] "Generic (PLEG): container finished" podID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerID="43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd" exitCode=0
Jul 10 19:13:31.159478 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:31.159134    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" event={"ID":"61245d75-9889-4b3f-ba6c-4b671b6607c9","Type":"ContainerDied","Data":"43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd"}
Jul 10 19:13:34.413785 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:34.413740    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:13:34.414226 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:34.414030    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:34.765392 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:34.765300    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 10 19:13:37.167932 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:37.167880    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:13:39.765686 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:39.765646    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 10 19:13:39.766104 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:39.765763    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:13:44.414294 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:44.414246    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:13:44.414807 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:44.414558    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:44.765486 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:44.765392    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 10 19:13:46.199616 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.199584    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:13:46.231044 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.231018    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"]
Jul 10 19:13:46.234501 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.234480    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.237009 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.236985    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 10 19:13:46.237120 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.237100    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 10 19:13:46.249847 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.249254    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"]
Jul 10 19:13:46.319698 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.319671    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kserve-provision-location\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.319796 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.319708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lzkz\" (UniqueName: \"kubernetes.io/projected/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-api-access-5lzkz\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.319796 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.319730    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.319875 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.319797    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-proxy-tls\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.319875 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.319837    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-cabundle-cert\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.420400 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.420367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kserve-provision-location\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.420556 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.420412    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lzkz\" (UniqueName: \"kubernetes.io/projected/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-api-access-5lzkz\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.420556 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.420448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.420556 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.420483    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-proxy-tls\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.420556 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.420518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-cabundle-cert\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.420779 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:13:46.420648    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 10 19:13:46.420779 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:13:46.420720    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-proxy-tls podName:5a130c7f-0bd5-44ba-a93d-bd395b7521eb nodeName:}" failed. No retries permitted until 2026-07-10 19:13:46.920698766 +0000 UTC m=+930.172346077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-proxy-tls") pod "isvc-logger-predictor-5bffdc48ff-f9khp" (UID: "5a130c7f-0bd5-44ba-a93d-bd395b7521eb") : secret "isvc-logger-predictor-serving-cert" not found
Jul 10 19:13:46.420896 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.420803    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kserve-provision-location\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.421220 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.421196    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.421297 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.421225    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-cabundle-cert\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.431774 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.431752    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lzkz\" (UniqueName: \"kubernetes.io/projected/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-api-access-5lzkz\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.924374 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.924340    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-proxy-tls\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:46.926824 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:46.926792    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-proxy-tls\") pod \"isvc-logger-predictor-5bffdc48ff-f9khp\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:47.146845 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:47.146811    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:47.281415 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:47.281381    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"]
Jul 10 19:13:47.286310 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:13:47.286285    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a130c7f_0bd5_44ba_a93d_bd395b7521eb.slice/crio-3d0ba4a28408b4c906eabcb78c3339e1bd3be9bba846db1045e8fc83d7a6c309 WatchSource:0}: Error finding container 3d0ba4a28408b4c906eabcb78c3339e1bd3be9bba846db1045e8fc83d7a6c309: Status 404 returned error can't find the container with id 3d0ba4a28408b4c906eabcb78c3339e1bd3be9bba846db1045e8fc83d7a6c309
Jul 10 19:13:47.839955 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:47.839895    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:13:48.232838 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:48.232754    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" event={"ID":"5a130c7f-0bd5-44ba-a93d-bd395b7521eb","Type":"ContainerStarted","Data":"db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7"}
Jul 10 19:13:48.232838 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:48.232821    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" event={"ID":"5a130c7f-0bd5-44ba-a93d-bd395b7521eb","Type":"ContainerStarted","Data":"3d0ba4a28408b4c906eabcb78c3339e1bd3be9bba846db1045e8fc83d7a6c309"}
Jul 10 19:13:49.765429 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:49.765387    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 10 19:13:51.247042 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:51.246961    2567 generic.go:356] "Generic (PLEG): container finished" podID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerID="db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7" exitCode=0
Jul 10 19:13:51.247419 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:51.247032    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" event={"ID":"5a130c7f-0bd5-44ba-a93d-bd395b7521eb","Type":"ContainerDied","Data":"db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7"}
Jul 10 19:13:52.253830 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:52.253797    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" event={"ID":"5a130c7f-0bd5-44ba-a93d-bd395b7521eb","Type":"ContainerStarted","Data":"2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee"}
Jul 10 19:13:52.254267 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:52.253866    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" event={"ID":"5a130c7f-0bd5-44ba-a93d-bd395b7521eb","Type":"ContainerStarted","Data":"87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a"}
Jul 10 19:13:52.254267 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:52.253895    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" event={"ID":"5a130c7f-0bd5-44ba-a93d-bd395b7521eb","Type":"ContainerStarted","Data":"707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4"}
Jul 10 19:13:52.254347 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:52.254301    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:52.254389 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:52.254355    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:52.254389 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:52.254379    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:52.255497 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:52.255471    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:13:52.256169 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:52.256143    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:52.277390 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:52.277341    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podStartSLOduration=2.684920726 podStartE2EDuration="6.277328307s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.592407581s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:13:52.273515942 +0000 UTC m=+935.525163272" watchObservedRunningTime="2026-07-10 19:13:52.277328307 +0000 UTC m=+935.528975635"
Jul 10 19:13:53.258413 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:53.258371    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:13:53.258801 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:53.258739    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:54.413966 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:54.413902    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 10 19:13:54.414463 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:54.414055    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:13:54.414463 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:54.414207    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:54.414463 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:54.414313    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:13:54.765553 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:54.765468    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 10 19:13:56.245618 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.245587    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:13:56.273276 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.273248    2567 generic.go:356] "Generic (PLEG): container finished" podID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerID="99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8" exitCode=0
Jul 10 19:13:56.273385 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.273306    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"
Jul 10 19:13:56.273385 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.273328    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" event={"ID":"61245d75-9889-4b3f-ba6c-4b671b6607c9","Type":"ContainerDied","Data":"99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8"}
Jul 10 19:13:56.273452 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.273412    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm" event={"ID":"61245d75-9889-4b3f-ba6c-4b671b6607c9","Type":"ContainerDied","Data":"bb6edcb5280af919b01fcb9bf7f32bf54f71b47185a3464c47c9d2adbb00ae82"}
Jul 10 19:13:56.273489 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.273459    2567 scope.go:117] "RemoveContainer" containerID="99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8"
Jul 10 19:13:56.281368 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.281345    2567 scope.go:117] "RemoveContainer" containerID="fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596"
Jul 10 19:13:56.288556 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.288536    2567 scope.go:117] "RemoveContainer" containerID="43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd"
Jul 10 19:13:56.295632 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.295617    2567 scope.go:117] "RemoveContainer" containerID="f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d"
Jul 10 19:13:56.302828 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.302813    2567 scope.go:117] "RemoveContainer" containerID="99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8"
Jul 10 19:13:56.303089 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:13:56.303072    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8\": container with ID starting with 99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8 not found: ID does not exist" containerID="99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8"
Jul 10 19:13:56.303153 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.303101    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8"} err="failed to get container status \"99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8\": rpc error: code = NotFound desc = could not find container \"99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8\": container with ID starting with 99b56878b94081cf13682717c6db37fdef96c8aee6739ef989d7285ae5ea9dc8 not found: ID does not exist"
Jul 10 19:13:56.303153 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.303118    2567 scope.go:117] "RemoveContainer" containerID="fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596"
Jul 10 19:13:56.303353 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:13:56.303335    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596\": container with ID starting with fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596 not found: ID does not exist" containerID="fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596"
Jul 10 19:13:56.303403 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.303360    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596"} err="failed to get container status \"fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596\": rpc error: code = NotFound desc = could not find container \"fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596\": container with ID starting with fbd23f1d890e4ef55277fed4182afdd41d6f14616be218fe4802e60a4c743596 not found: ID does not exist"
Jul 10 19:13:56.303403 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.303375    2567 scope.go:117] "RemoveContainer" containerID="43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd"
Jul 10 19:13:56.303583 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:13:56.303566    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd\": container with ID starting with 43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd not found: ID does not exist" containerID="43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd"
Jul 10 19:13:56.303645 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.303591    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd"} err="failed to get container status \"43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd\": rpc error: code = NotFound desc = could not find container \"43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd\": container with ID starting with 43784ef9b44fabdc2c9590ff3514018f690f4895d69792c9ae99173b898d39cd not found: ID does not exist"
Jul 10 19:13:56.303645 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.303612    2567 scope.go:117] "RemoveContainer" containerID="f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d"
Jul 10 19:13:56.303837 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:13:56.303823    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d\": container with ID starting with f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d not found: ID does not exist" containerID="f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d"
Jul 10 19:13:56.303879 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.303841    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d"} err="failed to get container status \"f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d\": rpc error: code = NotFound desc = could not find container \"f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d\": container with ID starting with f666666fb1ba3135f8182e381ccc7f7e2cad10d74ede33c333f6e5b285f5538d not found: ID does not exist"
Jul 10 19:13:56.406330 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.406277    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61245d75-9889-4b3f-ba6c-4b671b6607c9-kserve-provision-location\") pod \"61245d75-9889-4b3f-ba6c-4b671b6607c9\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") "
Jul 10 19:13:56.406416 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.406345    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzfg8\" (UniqueName: \"kubernetes.io/projected/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-api-access-zzfg8\") pod \"61245d75-9889-4b3f-ba6c-4b671b6607c9\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") "
Jul 10 19:13:56.406416 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.406373    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-cabundle-cert\") pod \"61245d75-9889-4b3f-ba6c-4b671b6607c9\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") "
Jul 10 19:13:56.406416 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.406398    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61245d75-9889-4b3f-ba6c-4b671b6607c9-proxy-tls\") pod \"61245d75-9889-4b3f-ba6c-4b671b6607c9\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") "
Jul 10 19:13:56.406525 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.406431    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-rbac-proxy-sar-config\") pod \"61245d75-9889-4b3f-ba6c-4b671b6607c9\" (UID: \"61245d75-9889-4b3f-ba6c-4b671b6607c9\") "
Jul 10 19:13:56.406611 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.406587    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61245d75-9889-4b3f-ba6c-4b671b6607c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61245d75-9889-4b3f-ba6c-4b671b6607c9" (UID: "61245d75-9889-4b3f-ba6c-4b671b6607c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:13:56.406767 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.406747    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "61245d75-9889-4b3f-ba6c-4b671b6607c9" (UID: "61245d75-9889-4b3f-ba6c-4b671b6607c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:13:56.406824 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.406784    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "61245d75-9889-4b3f-ba6c-4b671b6607c9" (UID: "61245d75-9889-4b3f-ba6c-4b671b6607c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:13:56.408346 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.408320    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61245d75-9889-4b3f-ba6c-4b671b6607c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61245d75-9889-4b3f-ba6c-4b671b6607c9" (UID: "61245d75-9889-4b3f-ba6c-4b671b6607c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:13:56.408421 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.408394    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-api-access-zzfg8" (OuterVolumeSpecName: "kube-api-access-zzfg8") pod "61245d75-9889-4b3f-ba6c-4b671b6607c9" (UID: "61245d75-9889-4b3f-ba6c-4b671b6607c9"). InnerVolumeSpecName "kube-api-access-zzfg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:13:56.507313 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.507281    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61245d75-9889-4b3f-ba6c-4b671b6607c9-kserve-provision-location\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:13:56.507313 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.507312    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzfg8\" (UniqueName: \"kubernetes.io/projected/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-api-access-zzfg8\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:13:56.507475 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.507322    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-cabundle-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:13:56.507475 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.507331    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61245d75-9889-4b3f-ba6c-4b671b6607c9-proxy-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:13:56.507475 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.507339    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61245d75-9889-4b3f-ba6c-4b671b6607c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:13:56.595950 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.595920    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"]
Jul 10 19:13:56.599492 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:56.599472    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-578cb45d67-d85mm"]
Jul 10 19:13:57.418724 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:57.418689    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" path="/var/lib/kubelet/pods/61245d75-9889-4b3f-ba6c-4b671b6607c9/volumes"
Jul 10 19:13:58.262602 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:58.262569    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:13:58.263214 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:58.263168    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:13:58.263543 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:13:58.263520    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:14:08.263957 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:08.263896    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:14:08.264422 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:08.264347    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:14:18.263250 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:18.263204    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:14:18.263733 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:18.263635    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:14:28.263889 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:28.263848    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:14:28.264321 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:28.264249    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:14:38.263222 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:38.263169    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:14:38.263628 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:38.263573    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:14:48.263614 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:48.263521    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:14:48.264102 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:48.264042    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:14:58.263243 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:58.263199    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:14:58.263709 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:14:58.263681    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:08.264355 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:08.264322    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:15:08.264775 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:08.264484    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:15:21.243012 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.242981    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-828h5_d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f/kserve-container/0.log"
Jul 10 19:15:21.434058 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.434023    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"]
Jul 10 19:15:21.434318 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.434289    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" containerID="cri-o://2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee" gracePeriod=30
Jul 10 19:15:21.434400 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.434284    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" containerID="cri-o://707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4" gracePeriod=30
Jul 10 19:15:21.434400 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.434303    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kube-rbac-proxy" containerID="cri-o://87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a" gracePeriod=30
Jul 10 19:15:21.467385 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.467360    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:15:21.533889 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.533817    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"]
Jul 10 19:15:21.534074 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.534047    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5" podUID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" containerName="kserve-container" containerID="cri-o://061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d" gracePeriod=30
Jul 10 19:15:21.534144 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.534095    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5" podUID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" containerName="kube-rbac-proxy" containerID="cri-o://de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8" gracePeriod=30
Jul 10 19:15:21.547797 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.547757    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:15:21.597718 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.597692    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"]
Jul 10 19:15:21.598189 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598174    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent"
Jul 10 19:15:21.598243 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598192    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent"
Jul 10 19:15:21.598243 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598215    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kube-rbac-proxy"
Jul 10 19:15:21.598243 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598224    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kube-rbac-proxy"
Jul 10 19:15:21.598243 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598239    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="storage-initializer"
Jul 10 19:15:21.598378 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598249    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="storage-initializer"
Jul 10 19:15:21.598378 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598260    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container"
Jul 10 19:15:21.598378 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598268    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container"
Jul 10 19:15:21.598378 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598348    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kube-rbac-proxy"
Jul 10 19:15:21.598378 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598361    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="kserve-container"
Jul 10 19:15:21.598378 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.598372    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="61245d75-9889-4b3f-ba6c-4b671b6607c9" containerName="agent"
Jul 10 19:15:21.601681 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.601663    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.604134 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.604114    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 10 19:15:21.604399 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.604384    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 10 19:15:21.612005 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.611984    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"]
Jul 10 19:15:21.634825 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.634784    2567 generic.go:356] "Generic (PLEG): container finished" podID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerID="87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a" exitCode=2
Jul 10 19:15:21.634955 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.634859    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" event={"ID":"5a130c7f-0bd5-44ba-a93d-bd395b7521eb","Type":"ContainerDied","Data":"87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a"}
Jul 10 19:15:21.636468 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.636450    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:15:21.683766 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.683729    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a12b378-89c2-4525-87b4-91b19d19bd66-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.683933 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.683788    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.683933 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.683845    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.684028 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.683926    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a12b378-89c2-4525-87b4-91b19d19bd66-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.684028 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.683990    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkzwc\" (UniqueName: \"kubernetes.io/projected/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-api-access-tkzwc\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.782298 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.782272    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:15:21.785359 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.785320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkzwc\" (UniqueName: \"kubernetes.io/projected/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-api-access-tkzwc\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.785427 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.785357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a12b378-89c2-4525-87b4-91b19d19bd66-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.785427 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.785381    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.785427 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.785415    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.785563 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.785452    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a12b378-89c2-4525-87b4-91b19d19bd66-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.785983 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.785957    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a12b378-89c2-4525-87b4-91b19d19bd66-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.786189 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.786169    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.786259 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.786187    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.787534 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.787517    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a12b378-89c2-4525-87b4-91b19d19bd66-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.793690 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.793672    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkzwc\" (UniqueName: \"kubernetes.io/projected/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-api-access-tkzwc\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.826940 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.826897    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:15:21.886190 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.886165    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-rbac-proxy-sar-config\") pod \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") "
Jul 10 19:15:21.886265 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.886198    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2td7\" (UniqueName: \"kubernetes.io/projected/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-api-access-j2td7\") pod \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") "
Jul 10 19:15:21.886265 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.886216    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-proxy-tls\") pod \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\" (UID: \"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f\") "
Jul 10 19:15:21.886540 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.886516    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" (UID: "d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:15:21.888106 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.888083    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-api-access-j2td7" (OuterVolumeSpecName: "kube-api-access-j2td7") pod "d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" (UID: "d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f"). InnerVolumeSpecName "kube-api-access-j2td7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:15:21.888247 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.888225    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" (UID: "d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:15:21.920523 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.920496    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:15:21.987193 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.987164    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:15:21.987193 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.987191    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2td7\" (UniqueName: \"kubernetes.io/projected/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-kube-api-access-j2td7\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:15:21.987375 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:21.987202    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f-proxy-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:15:22.049938 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.049899    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"]
Jul 10 19:15:22.051706 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:15:22.051679    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a12b378_89c2_4525_87b4_91b19d19bd66.slice/crio-a4e2986078146d7264452cd0139407447c6a033ca11d0e252dbb07c3bf07d82c WatchSource:0}: Error finding container a4e2986078146d7264452cd0139407447c6a033ca11d0e252dbb07c3bf07d82c: Status 404 returned error can't find the container with id a4e2986078146d7264452cd0139407447c6a033ca11d0e252dbb07c3bf07d82c
Jul 10 19:15:22.640885 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.640849    2567 generic.go:356] "Generic (PLEG): container finished" podID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" containerID="de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8" exitCode=2
Jul 10 19:15:22.640885 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.640880    2567 generic.go:356] "Generic (PLEG): container finished" podID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" containerID="061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d" exitCode=2
Jul 10 19:15:22.641346 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.640927    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"
Jul 10 19:15:22.641346 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.640947    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5" event={"ID":"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f","Type":"ContainerDied","Data":"de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8"}
Jul 10 19:15:22.641346 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.641014    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5" event={"ID":"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f","Type":"ContainerDied","Data":"061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d"}
Jul 10 19:15:22.641346 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.641038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5" event={"ID":"d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f","Type":"ContainerDied","Data":"eb85df886a0ed3b6ca173adac2e870b9f613aef87d742b945c44b5cdd23d77ec"}
Jul 10 19:15:22.641346 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.641067    2567 scope.go:117] "RemoveContainer" containerID="de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8"
Jul 10 19:15:22.642736 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.642711    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" event={"ID":"7a12b378-89c2-4525-87b4-91b19d19bd66","Type":"ContainerStarted","Data":"653a5be7f518e39467cb0c651a43354054902577e261aa3d0e10b8e5bdcc3103"}
Jul 10 19:15:22.642864 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.642764    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" event={"ID":"7a12b378-89c2-4525-87b4-91b19d19bd66","Type":"ContainerStarted","Data":"a4e2986078146d7264452cd0139407447c6a033ca11d0e252dbb07c3bf07d82c"}
Jul 10 19:15:22.651535 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.651401    2567 scope.go:117] "RemoveContainer" containerID="061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d"
Jul 10 19:15:22.659186 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.659166    2567 scope.go:117] "RemoveContainer" containerID="de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8"
Jul 10 19:15:22.659449 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:15:22.659426    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8\": container with ID starting with de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8 not found: ID does not exist" containerID="de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8"
Jul 10 19:15:22.659521 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.659457    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8"} err="failed to get container status \"de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8\": rpc error: code = NotFound desc = could not find container \"de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8\": container with ID starting with de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8 not found: ID does not exist"
Jul 10 19:15:22.659521 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.659473    2567 scope.go:117] "RemoveContainer" containerID="061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d"
Jul 10 19:15:22.659711 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:15:22.659693    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d\": container with ID starting with 061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d not found: ID does not exist" containerID="061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d"
Jul 10 19:15:22.659774 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.659717    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d"} err="failed to get container status \"061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d\": rpc error: code = NotFound desc = could not find container \"061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d\": container with ID starting with 061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d not found: ID does not exist"
Jul 10 19:15:22.659774 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.659734    2567 scope.go:117] "RemoveContainer" containerID="de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8"
Jul 10 19:15:22.659979 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.659959    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8"} err="failed to get container status \"de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8\": rpc error: code = NotFound desc = could not find container \"de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8\": container with ID starting with de5e53b7e84c2e36ec58f9068cd05af15a4505f43397d542bc145cedeec2b6a8 not found: ID does not exist"
Jul 10 19:15:22.660058 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.659979    2567 scope.go:117] "RemoveContainer" containerID="061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d"
Jul 10 19:15:22.660195 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.660177    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d"} err="failed to get container status \"061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d\": rpc error: code = NotFound desc = could not find container \"061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d\": container with ID starting with 061c8e1c2683f38085911478e6b423c2730f63074129b766fac709eaa27ea55d not found: ID does not exist"
Jul 10 19:15:22.676208 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.676174    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"]
Jul 10 19:15:22.680414 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:22.680391    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-828h5"]
Jul 10 19:15:23.259521 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:23.259481    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 10 19:15:23.420200 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:23.420166    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" path="/var/lib/kubelet/pods/d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f/volumes"
Jul 10 19:15:25.658061 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:25.658032    2567 generic.go:356] "Generic (PLEG): container finished" podID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerID="707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4" exitCode=0
Jul 10 19:15:25.658381 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:25.658081    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" event={"ID":"5a130c7f-0bd5-44ba-a93d-bd395b7521eb","Type":"ContainerDied","Data":"707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4"}
Jul 10 19:15:28.258933 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:28.258864    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 10 19:15:28.263215 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:28.263184    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:15:28.263560 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:28.263535    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:29.674656 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:29.674624    2567 generic.go:356] "Generic (PLEG): container finished" podID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerID="653a5be7f518e39467cb0c651a43354054902577e261aa3d0e10b8e5bdcc3103" exitCode=0
Jul 10 19:15:29.675035 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:29.674674    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" event={"ID":"7a12b378-89c2-4525-87b4-91b19d19bd66","Type":"ContainerDied","Data":"653a5be7f518e39467cb0c651a43354054902577e261aa3d0e10b8e5bdcc3103"}
Jul 10 19:15:33.258928 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:33.258862    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 10 19:15:33.259349 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:33.259038    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:15:38.259266 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:38.259219    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 10 19:15:38.263400 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:38.263367    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:15:38.263744 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:38.263718    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:43.259350 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:43.259292    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 10 19:15:48.259147 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:48.259107    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 10 19:15:48.263557 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:48.263522    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:15:48.263700 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:48.263677    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:15:48.263872 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:48.263838    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:48.264017 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:48.263998    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:15:51.433315 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.433285    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:15:51.595201 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.595176    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:15:51.733244 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.733163    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kserve-provision-location\") pod \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") "
Jul 10 19:15:51.733244 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.733231    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-rbac-proxy-sar-config\") pod \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") "
Jul 10 19:15:51.733464 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.733291    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-cabundle-cert\") pod \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") "
Jul 10 19:15:51.733464 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.733373    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-proxy-tls\") pod \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") "
Jul 10 19:15:51.733464 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.733401    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lzkz\" (UniqueName: \"kubernetes.io/projected/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-api-access-5lzkz\") pod \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\" (UID: \"5a130c7f-0bd5-44ba-a93d-bd395b7521eb\") "
Jul 10 19:15:51.733679 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.733651    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a130c7f-0bd5-44ba-a93d-bd395b7521eb" (UID: "5a130c7f-0bd5-44ba-a93d-bd395b7521eb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:15:51.733763 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.733649    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a130c7f-0bd5-44ba-a93d-bd395b7521eb" (UID: "5a130c7f-0bd5-44ba-a93d-bd395b7521eb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:15:51.733763 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.733682    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a130c7f-0bd5-44ba-a93d-bd395b7521eb" (UID: "5a130c7f-0bd5-44ba-a93d-bd395b7521eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:15:51.735761 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.735727    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a130c7f-0bd5-44ba-a93d-bd395b7521eb" (UID: "5a130c7f-0bd5-44ba-a93d-bd395b7521eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:15:51.735878 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.735840    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-api-access-5lzkz" (OuterVolumeSpecName: "kube-api-access-5lzkz") pod "5a130c7f-0bd5-44ba-a93d-bd395b7521eb" (UID: "5a130c7f-0bd5-44ba-a93d-bd395b7521eb"). InnerVolumeSpecName "kube-api-access-5lzkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:15:51.777991 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.777938    2567 generic.go:356] "Generic (PLEG): container finished" podID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerID="2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee" exitCode=0
Jul 10 19:15:51.778132 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.777979    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" event={"ID":"5a130c7f-0bd5-44ba-a93d-bd395b7521eb","Type":"ContainerDied","Data":"2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee"}
Jul 10 19:15:51.778132 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.778035    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"
Jul 10 19:15:51.778132 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.778069    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp" event={"ID":"5a130c7f-0bd5-44ba-a93d-bd395b7521eb","Type":"ContainerDied","Data":"3d0ba4a28408b4c906eabcb78c3339e1bd3be9bba846db1045e8fc83d7a6c309"}
Jul 10 19:15:51.778132 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.778122    2567 scope.go:117] "RemoveContainer" containerID="2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee"
Jul 10 19:15:51.788817 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.788795    2567 scope.go:117] "RemoveContainer" containerID="87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a"
Jul 10 19:15:51.799989 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.799875    2567 scope.go:117] "RemoveContainer" containerID="707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4"
Jul 10 19:15:51.803807 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.803786    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"]
Jul 10 19:15:51.806568 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.806547    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5bffdc48ff-f9khp"]
Jul 10 19:15:51.808709 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.808695    2567 scope.go:117] "RemoveContainer" containerID="db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7"
Jul 10 19:15:51.815599 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.815583    2567 scope.go:117] "RemoveContainer" containerID="2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee"
Jul 10 19:15:51.815865 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:15:51.815847    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee\": container with ID starting with 2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee not found: ID does not exist" containerID="2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee"
Jul 10 19:15:51.815927 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.815873    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee"} err="failed to get container status \"2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee\": rpc error: code = NotFound desc = could not find container \"2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee\": container with ID starting with 2fae338c54824083501531ac79dd260a7722bdb7bd8986a5c98319f81d0ed2ee not found: ID does not exist"
Jul 10 19:15:51.815927 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.815890    2567 scope.go:117] "RemoveContainer" containerID="87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a"
Jul 10 19:15:51.816148 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:15:51.816133    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a\": container with ID starting with 87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a not found: ID does not exist" containerID="87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a"
Jul 10 19:15:51.816189 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.816164    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a"} err="failed to get container status \"87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a\": rpc error: code = NotFound desc = could not find container \"87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a\": container with ID starting with 87d282ef996363e04494b75d790cc9bf2378a69d39873cf8d17b6a06c0bd428a not found: ID does not exist"
Jul 10 19:15:51.816189 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.816178    2567 scope.go:117] "RemoveContainer" containerID="707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4"
Jul 10 19:15:51.816411 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:15:51.816396    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4\": container with ID starting with 707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4 not found: ID does not exist" containerID="707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4"
Jul 10 19:15:51.816453 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.816415    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4"} err="failed to get container status \"707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4\": rpc error: code = NotFound desc = could not find container \"707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4\": container with ID starting with 707dc578b7a7b437506e468613160521b2442910a3e25270b5fbc188475b50c4 not found: ID does not exist"
Jul 10 19:15:51.816453 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.816430    2567 scope.go:117] "RemoveContainer" containerID="db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7"
Jul 10 19:15:51.816651 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:15:51.816632    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7\": container with ID starting with db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7 not found: ID does not exist" containerID="db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7"
Jul 10 19:15:51.816689 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.816658    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7"} err="failed to get container status \"db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7\": rpc error: code = NotFound desc = could not find container \"db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7\": container with ID starting with db3b26e5a44913df6a415fbc4a36839bae98246a57db21d2f29cb9bd70f08dc7 not found: ID does not exist"
Jul 10 19:15:51.834194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.834173    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-cabundle-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:15:51.834194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.834192    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-proxy-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:15:51.834291 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.834201    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lzkz\" (UniqueName: \"kubernetes.io/projected/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-api-access-5lzkz\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:15:51.834291 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.834210    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kserve-provision-location\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:15:51.834291 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:51.834218    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a130c7f-0bd5-44ba-a93d-bd395b7521eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:15:52.422113 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:52.422079    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:15:53.418452 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:15:53.418419    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" path="/var/lib/kubelet/pods/5a130c7f-0bd5-44ba-a93d-bd395b7521eb/volumes"
Jul 10 19:16:40.201666 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:16:40.201615    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:17:57.335070 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:17:57.335039    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" event={"ID":"7a12b378-89c2-4525-87b4-91b19d19bd66","Type":"ContainerStarted","Data":"9f21e32647f40874416f945bc6a4aa31e158bb82ad23272f7e9797e774f5a1a4"}
Jul 10 19:17:57.335541 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:17:57.335095    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" event={"ID":"7a12b378-89c2-4525-87b4-91b19d19bd66","Type":"ContainerStarted","Data":"ae50e0d79dec7a5dcb647e3313a27f3dd1c6ff69a5f0e7c9711c676ad7f657f3"}
Jul 10 19:17:57.335541 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:17:57.335216    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:17:57.368015 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:17:57.367956    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" podStartSLOduration=2.068327152 podStartE2EDuration="2m36.36793589s" totalImagesPullingTime="2m27.153000292s" totalInitContainerRuntime="7.146608446s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:17:57.366618855 +0000 UTC m=+1180.618266208" watchObservedRunningTime="2026-07-10 19:17:57.36793589 +0000 UTC m=+1180.619583219"
Jul 10 19:17:58.339619 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:17:58.339584    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:17:58.340705 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:17:58.340676    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 10 19:17:59.343378 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:17:59.343340    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 10 19:18:04.348387 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:04.348359    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:18:04.348998 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:04.348935    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 10 19:18:14.350061 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:14.350033    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:18:21.722938 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:21.722893    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"]
Jul 10 19:18:21.723467 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:21.723162    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kserve-container" containerID="cri-o://ae50e0d79dec7a5dcb647e3313a27f3dd1c6ff69a5f0e7c9711c676ad7f657f3" gracePeriod=30
Jul 10 19:18:21.723467 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:21.723201    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kube-rbac-proxy" containerID="cri-o://9f21e32647f40874416f945bc6a4aa31e158bb82ad23272f7e9797e774f5a1a4" gracePeriod=30
Jul 10 19:18:21.777526 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:21.777499    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:18:22.439529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.439501    2567 generic.go:356] "Generic (PLEG): container finished" podID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerID="9f21e32647f40874416f945bc6a4aa31e158bb82ad23272f7e9797e774f5a1a4" exitCode=2
Jul 10 19:18:22.439529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.439525    2567 generic.go:356] "Generic (PLEG): container finished" podID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerID="ae50e0d79dec7a5dcb647e3313a27f3dd1c6ff69a5f0e7c9711c676ad7f657f3" exitCode=0
Jul 10 19:18:22.439676 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.439574    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" event={"ID":"7a12b378-89c2-4525-87b4-91b19d19bd66","Type":"ContainerDied","Data":"9f21e32647f40874416f945bc6a4aa31e158bb82ad23272f7e9797e774f5a1a4"}
Jul 10 19:18:22.439676 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.439628    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" event={"ID":"7a12b378-89c2-4525-87b4-91b19d19bd66","Type":"ContainerDied","Data":"ae50e0d79dec7a5dcb647e3313a27f3dd1c6ff69a5f0e7c9711c676ad7f657f3"}
Jul 10 19:18:22.595371 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.595349    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:18:22.725657 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.725584    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-rbac-proxy-sar-config\") pod \"7a12b378-89c2-4525-87b4-91b19d19bd66\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") "
Jul 10 19:18:22.725657 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.725636    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a12b378-89c2-4525-87b4-91b19d19bd66-proxy-tls\") pod \"7a12b378-89c2-4525-87b4-91b19d19bd66\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") "
Jul 10 19:18:22.726152 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.725702    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkzwc\" (UniqueName: \"kubernetes.io/projected/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-api-access-tkzwc\") pod \"7a12b378-89c2-4525-87b4-91b19d19bd66\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") "
Jul 10 19:18:22.726152 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.725729    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-cabundle-cert\") pod \"7a12b378-89c2-4525-87b4-91b19d19bd66\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") "
Jul 10 19:18:22.726152 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.725772    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a12b378-89c2-4525-87b4-91b19d19bd66-kserve-provision-location\") pod \"7a12b378-89c2-4525-87b4-91b19d19bd66\" (UID: \"7a12b378-89c2-4525-87b4-91b19d19bd66\") "
Jul 10 19:18:22.726152 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.725894    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a12b378-89c2-4525-87b4-91b19d19bd66" (UID: "7a12b378-89c2-4525-87b4-91b19d19bd66"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:18:22.726152 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.726001    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:18:22.726152 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.726128    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a12b378-89c2-4525-87b4-91b19d19bd66" (UID: "7a12b378-89c2-4525-87b4-91b19d19bd66"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:18:22.727691 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.727670    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-api-access-tkzwc" (OuterVolumeSpecName: "kube-api-access-tkzwc") pod "7a12b378-89c2-4525-87b4-91b19d19bd66" (UID: "7a12b378-89c2-4525-87b4-91b19d19bd66"). InnerVolumeSpecName "kube-api-access-tkzwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:18:22.727788 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.727675    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a12b378-89c2-4525-87b4-91b19d19bd66-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a12b378-89c2-4525-87b4-91b19d19bd66" (UID: "7a12b378-89c2-4525-87b4-91b19d19bd66"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:18:22.729395 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.729368    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a12b378-89c2-4525-87b4-91b19d19bd66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a12b378-89c2-4525-87b4-91b19d19bd66" (UID: "7a12b378-89c2-4525-87b4-91b19d19bd66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:18:22.827297 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.827268    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a12b378-89c2-4525-87b4-91b19d19bd66-proxy-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:18:22.827297 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.827296    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkzwc\" (UniqueName: \"kubernetes.io/projected/7a12b378-89c2-4525-87b4-91b19d19bd66-kube-api-access-tkzwc\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:18:22.827420 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.827307    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a12b378-89c2-4525-87b4-91b19d19bd66-cabundle-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:18:22.827420 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:22.827315    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a12b378-89c2-4525-87b4-91b19d19bd66-kserve-provision-location\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:18:23.445337 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:23.445305    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb" event={"ID":"7a12b378-89c2-4525-87b4-91b19d19bd66","Type":"ContainerDied","Data":"a4e2986078146d7264452cd0139407447c6a033ca11d0e252dbb07c3bf07d82c"}
Jul 10 19:18:23.445337 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:23.445334    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"
Jul 10 19:18:23.445534 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:23.445376    2567 scope.go:117] "RemoveContainer" containerID="9f21e32647f40874416f945bc6a4aa31e158bb82ad23272f7e9797e774f5a1a4"
Jul 10 19:18:23.453281 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:23.453264    2567 scope.go:117] "RemoveContainer" containerID="ae50e0d79dec7a5dcb647e3313a27f3dd1c6ff69a5f0e7c9711c676ad7f657f3"
Jul 10 19:18:23.460223 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:23.460206    2567 scope.go:117] "RemoveContainer" containerID="653a5be7f518e39467cb0c651a43354054902577e261aa3d0e10b8e5bdcc3103"
Jul 10 19:18:23.463680 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:23.463660    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"]
Jul 10 19:18:23.467691 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:23.467668    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-sqzpb"]
Jul 10 19:18:25.419257 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:25.419222    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" path="/var/lib/kubelet/pods/7a12b378-89c2-4525-87b4-91b19d19bd66/volumes"
Jul 10 19:18:26.965151 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:26.965116    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:18:27.016376 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016343    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"]
Jul 10 19:18:27.016730 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016712    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" containerName="kube-rbac-proxy"
Jul 10 19:18:27.016819 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016733    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" containerName="kube-rbac-proxy"
Jul 10 19:18:27.016819 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016749    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" containerName="kserve-container"
Jul 10 19:18:27.016819 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016756    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" containerName="kserve-container"
Jul 10 19:18:27.016819 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016784    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kube-rbac-proxy"
Jul 10 19:18:27.016819 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016793    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kube-rbac-proxy"
Jul 10 19:18:27.016819 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016811    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container"
Jul 10 19:18:27.016819 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016820    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016840    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="storage-initializer"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016850    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="storage-initializer"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016861    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="storage-initializer"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016870    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="storage-initializer"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016879    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kserve-container"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016887    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kserve-container"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016901    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kube-rbac-proxy"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016936    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kube-rbac-proxy"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016947    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.016954    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.017037    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kube-rbac-proxy"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.017048    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="agent"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.017059    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a130c7f-0bd5-44ba-a93d-bd395b7521eb" containerName="kserve-container"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.017070    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kube-rbac-proxy"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.017079    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a12b378-89c2-4525-87b4-91b19d19bd66" containerName="kserve-container"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.017087    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" containerName="kserve-container"
Jul 10 19:18:27.017188 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.017098    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4bd1df1-ea5a-489a-aaf1-cdd6ac3b295f" containerName="kube-rbac-proxy"
Jul 10 19:18:27.034080 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.034056    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"]
Jul 10 19:18:27.034198 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.034185    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.038806 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.038785    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 10 19:18:27.038951 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.038893    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z7zjv\""
Jul 10 19:18:27.039026 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.038791    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:18:27.039077 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.039064    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 10 19:18:27.039157 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.039122    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 19:18:27.039275 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.039173    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 19:18:27.039723 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.039684    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 10 19:18:27.162386 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.162358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.162511 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.162405    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa0fc2e-9f00-4894-a334-f589f718a7dd-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.162511 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.162437    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.162644 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.162505    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2sc\" (UniqueName: \"kubernetes.io/projected/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-api-access-6w2sc\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.162644 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.162541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.263182 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.263115    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.263182 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.263176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa0fc2e-9f00-4894-a334-f589f718a7dd-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.263366 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.263216    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.263366 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.263246    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2sc\" (UniqueName: \"kubernetes.io/projected/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-api-access-6w2sc\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.263366 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.263272    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.263529 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:18:27.263360    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 10 19:18:27.263529 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:18:27.263425    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa0fc2e-9f00-4894-a334-f589f718a7dd-proxy-tls podName:8aa0fc2e-9f00-4894-a334-f589f718a7dd nodeName:}" failed. No retries permitted until 2026-07-10 19:18:27.763404882 +0000 UTC m=+1211.015052206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8aa0fc2e-9f00-4894-a334-f589f718a7dd-proxy-tls") pod "isvc-autogluon-v2-predictor-67b849d456-5xpnn" (UID: "8aa0fc2e-9f00-4894-a334-f589f718a7dd") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 10 19:18:27.263674 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.263654    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.263851 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.263831    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.263891 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.263832    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.271948 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.271900    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2sc\" (UniqueName: \"kubernetes.io/projected/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-api-access-6w2sc\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.563780 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.563751    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:18:27.768473 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.768434    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa0fc2e-9f00-4894-a334-f589f718a7dd-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.770777 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.770755    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa0fc2e-9f00-4894-a334-f589f718a7dd-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xpnn\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:27.945007 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:27.944925    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:18:28.102238 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:28.102167    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"]
Jul 10 19:18:28.104476 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:18:28.104444    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa0fc2e_9f00_4894_a334_f589f718a7dd.slice/crio-0d0cef85652cd5a01066ae8fcb28119d454da0feb81d00b8f0c3184aaf79fdb3 WatchSource:0}: Error finding container 0d0cef85652cd5a01066ae8fcb28119d454da0feb81d00b8f0c3184aaf79fdb3: Status 404 returned error can't find the container with id 0d0cef85652cd5a01066ae8fcb28119d454da0feb81d00b8f0c3184aaf79fdb3
Jul 10 19:18:28.468508 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:28.468417    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" event={"ID":"8aa0fc2e-9f00-4894-a334-f589f718a7dd","Type":"ContainerStarted","Data":"0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f"}
Jul 10 19:18:28.468508 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:18:28.468478    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" event={"ID":"8aa0fc2e-9f00-4894-a334-f589f718a7dd","Type":"ContainerStarted","Data":"0d0cef85652cd5a01066ae8fcb28119d454da0feb81d00b8f0c3184aaf79fdb3"}
Jul 10 19:19:02.598818 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:19:02.598781    2567 generic.go:356] "Generic (PLEG): container finished" podID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerID="0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f" exitCode=0
Jul 10 19:19:02.598818 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:19:02.598819    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" event={"ID":"8aa0fc2e-9f00-4894-a334-f589f718a7dd","Type":"ContainerDied","Data":"0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f"}
Jul 10 19:19:03.605017 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:19:03.604982    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" event={"ID":"8aa0fc2e-9f00-4894-a334-f589f718a7dd","Type":"ContainerStarted","Data":"a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc"}
Jul 10 19:19:03.605498 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:19:03.605053    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" event={"ID":"8aa0fc2e-9f00-4894-a334-f589f718a7dd","Type":"ContainerStarted","Data":"66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0"}
Jul 10 19:19:03.605498 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:19:03.605274    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:19:03.626128 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:19:03.626081    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" podStartSLOduration=3.872612352 podStartE2EDuration="37.626066969s" totalImagesPullingTime="0s" totalInitContainerRuntime="33.753454617s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:19:03.625021346 +0000 UTC m=+1246.876668674" watchObservedRunningTime="2026-07-10 19:19:03.626066969 +0000 UTC m=+1246.877714299"
Jul 10 19:19:04.609979 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:19:04.609936    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:19:10.621102 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:19:10.621020    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:20:41.418923 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:41.418869    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:20:47.245483 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:47.245449    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"]
Jul 10 19:20:47.246015 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:47.245685    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerName="kserve-container" containerID="cri-o://66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0" gracePeriod=30
Jul 10 19:20:47.246015 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:47.245744    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerName="kube-rbac-proxy" containerID="cri-o://a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc" gracePeriod=30
Jul 10 19:20:47.322650 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:47.322618    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:20:47.999040 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:47.999018    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:20:48.021425 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.021397    2567 generic.go:356] "Generic (PLEG): container finished" podID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerID="a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc" exitCode=2
Jul 10 19:20:48.021425 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.021425    2567 generic.go:356] "Generic (PLEG): container finished" podID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerID="66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0" exitCode=0
Jul 10 19:20:48.021629 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.021472    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"
Jul 10 19:20:48.021629 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.021530    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" event={"ID":"8aa0fc2e-9f00-4894-a334-f589f718a7dd","Type":"ContainerDied","Data":"a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc"}
Jul 10 19:20:48.021629 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.021576    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" event={"ID":"8aa0fc2e-9f00-4894-a334-f589f718a7dd","Type":"ContainerDied","Data":"66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0"}
Jul 10 19:20:48.021629 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.021602    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn" event={"ID":"8aa0fc2e-9f00-4894-a334-f589f718a7dd","Type":"ContainerDied","Data":"0d0cef85652cd5a01066ae8fcb28119d454da0feb81d00b8f0c3184aaf79fdb3"}
Jul 10 19:20:48.021629 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.021629    2567 scope.go:117] "RemoveContainer" containerID="a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc"
Jul 10 19:20:48.029689 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.029673    2567 scope.go:117] "RemoveContainer" containerID="66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0"
Jul 10 19:20:48.036732 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.036709    2567 scope.go:117] "RemoveContainer" containerID="0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f"
Jul 10 19:20:48.043460 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.043441    2567 scope.go:117] "RemoveContainer" containerID="a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc"
Jul 10 19:20:48.043704 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:20:48.043686    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc\": container with ID starting with a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc not found: ID does not exist" containerID="a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc"
Jul 10 19:20:48.043742 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.043715    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc"} err="failed to get container status \"a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc\": rpc error: code = NotFound desc = could not find container \"a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc\": container with ID starting with a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc not found: ID does not exist"
Jul 10 19:20:48.043742 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.043738    2567 scope.go:117] "RemoveContainer" containerID="66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0"
Jul 10 19:20:48.044010 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:20:48.043992    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0\": container with ID starting with 66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0 not found: ID does not exist" containerID="66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0"
Jul 10 19:20:48.044054 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.044017    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0"} err="failed to get container status \"66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0\": rpc error: code = NotFound desc = could not find container \"66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0\": container with ID starting with 66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0 not found: ID does not exist"
Jul 10 19:20:48.044054 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.044032    2567 scope.go:117] "RemoveContainer" containerID="0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f"
Jul 10 19:20:48.044245 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:20:48.044229    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f\": container with ID starting with 0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f not found: ID does not exist" containerID="0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f"
Jul 10 19:20:48.044290 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.044247    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f"} err="failed to get container status \"0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f\": rpc error: code = NotFound desc = could not find container \"0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f\": container with ID starting with 0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f not found: ID does not exist"
Jul 10 19:20:48.044290 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.044259    2567 scope.go:117] "RemoveContainer" containerID="a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc"
Jul 10 19:20:48.044452 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.044432    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc"} err="failed to get container status \"a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc\": rpc error: code = NotFound desc = could not find container \"a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc\": container with ID starting with a0848a18b9f5102de333df2e575453a3bcb5ff4a99029210b1eaea12fd0e54cc not found: ID does not exist"
Jul 10 19:20:48.044497 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.044453    2567 scope.go:117] "RemoveContainer" containerID="66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0"
Jul 10 19:20:48.044664 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.044647    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0"} err="failed to get container status \"66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0\": rpc error: code = NotFound desc = could not find container \"66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0\": container with ID starting with 66b02f8def511bae069145f05f932e917429b9dd1acecbab981f366124c519d0 not found: ID does not exist"
Jul 10 19:20:48.044707 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.044665    2567 scope.go:117] "RemoveContainer" containerID="0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f"
Jul 10 19:20:48.044841 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.044826    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f"} err="failed to get container status \"0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f\": rpc error: code = NotFound desc = could not find container \"0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f\": container with ID starting with 0d858298d1323184d48524d6c425e99e8674a6688a960d4224c78e2be39d331f not found: ID does not exist"
Jul 10 19:20:48.108234 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.108175    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-cabundle-cert\") pod \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") "
Jul 10 19:20:48.108234 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.108201    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-rbac-proxy-sar-config\") pod \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") "
Jul 10 19:20:48.108234 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.108233    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa0fc2e-9f00-4894-a334-f589f718a7dd-proxy-tls\") pod \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") "
Jul 10 19:20:48.108459 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.108263    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w2sc\" (UniqueName: \"kubernetes.io/projected/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-api-access-6w2sc\") pod \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") "
Jul 10 19:20:48.108459 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.108314    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kserve-provision-location\") pod \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\" (UID: \"8aa0fc2e-9f00-4894-a334-f589f718a7dd\") "
Jul 10 19:20:48.108570 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.108516    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8aa0fc2e-9f00-4894-a334-f589f718a7dd" (UID: "8aa0fc2e-9f00-4894-a334-f589f718a7dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:20:48.108665 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.108552    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8aa0fc2e-9f00-4894-a334-f589f718a7dd" (UID: "8aa0fc2e-9f00-4894-a334-f589f718a7dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:20:48.110279 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.110260    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa0fc2e-9f00-4894-a334-f589f718a7dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8aa0fc2e-9f00-4894-a334-f589f718a7dd" (UID: "8aa0fc2e-9f00-4894-a334-f589f718a7dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:20:48.110371 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.110300    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-api-access-6w2sc" (OuterVolumeSpecName: "kube-api-access-6w2sc") pod "8aa0fc2e-9f00-4894-a334-f589f718a7dd" (UID: "8aa0fc2e-9f00-4894-a334-f589f718a7dd"). InnerVolumeSpecName "kube-api-access-6w2sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:20:48.112630 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.112602    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8aa0fc2e-9f00-4894-a334-f589f718a7dd" (UID: "8aa0fc2e-9f00-4894-a334-f589f718a7dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:20:48.208953 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.208900    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w2sc\" (UniqueName: \"kubernetes.io/projected/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-api-access-6w2sc\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.208953 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.208948    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kserve-provision-location\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.208953 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.208958    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-cabundle-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.209137 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.208967    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa0fc2e-9f00-4894-a334-f589f718a7dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.209137 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.208975    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa0fc2e-9f00-4894-a334-f589f718a7dd-proxy-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.343442 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.343414    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"]
Jul 10 19:20:48.346921 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:48.346886    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xpnn"]
Jul 10 19:20:49.418058 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:49.418021    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" path="/var/lib/kubelet/pods/8aa0fc2e-9f00-4894-a334-f589f718a7dd/volumes"
Jul 10 19:20:52.296360 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.296322    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:20:52.357812 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.357780    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"]
Jul 10 19:20:52.358194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.358167    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerName="storage-initializer"
Jul 10 19:20:52.358194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.358189    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerName="storage-initializer"
Jul 10 19:20:52.358350 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.358206    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerName="kserve-container"
Jul 10 19:20:52.358350 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.358211    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerName="kserve-container"
Jul 10 19:20:52.358350 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.358218    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerName="kube-rbac-proxy"
Jul 10 19:20:52.358350 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.358224    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerName="kube-rbac-proxy"
Jul 10 19:20:52.358350 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.358297    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerName="kube-rbac-proxy"
Jul 10 19:20:52.358350 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.358314    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa0fc2e-9f00-4894-a334-f589f718a7dd" containerName="kserve-container"
Jul 10 19:20:52.361380 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.361365    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.363865 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.363845    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 19:20:52.364121 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.364102    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 10 19:20:52.364224 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.364200    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 10 19:20:52.364319 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.364230    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 19:20:52.364376 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.364318    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 10 19:20:52.364858 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.364845    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:20:52.365008 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.364991    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z7zjv\""
Jul 10 19:20:52.371711 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.371689    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"]
Jul 10 19:20:52.405247 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.405224    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:20:52.443613 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.443590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.443705 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.443633    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1ab6420-329d-4ad7-9529-c68ef65af108-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.443705 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.443677    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1ab6420-329d-4ad7-9529-c68ef65af108-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.443783 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.443739    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfq64\" (UniqueName: \"kubernetes.io/projected/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-api-access-cfq64\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.443783 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.443772    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.545126 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.545087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.545422 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.545147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1ab6420-329d-4ad7-9529-c68ef65af108-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.545422 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.545183    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1ab6420-329d-4ad7-9529-c68ef65af108-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.545422 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.545209    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfq64\" (UniqueName: \"kubernetes.io/projected/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-api-access-cfq64\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.545422 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.545236    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.545700 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.545563    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1ab6420-329d-4ad7-9529-c68ef65af108-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.545808 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.545781    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.545943 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.545901    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.547536 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.547496    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1ab6420-329d-4ad7-9529-c68ef65af108-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.559152 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.559125    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfq64\" (UniqueName: \"kubernetes.io/projected/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-api-access-cfq64\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.672843 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.672809    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:52.797765 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:52.797603    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"]
Jul 10 19:20:52.800364 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:20:52.800338    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ab6420_329d_4ad7_9529_c68ef65af108.slice/crio-460fb597e85143805e711581d6bbef25dd89247f88fdfb052cf31840660889be WatchSource:0}: Error finding container 460fb597e85143805e711581d6bbef25dd89247f88fdfb052cf31840660889be: Status 404 returned error can't find the container with id 460fb597e85143805e711581d6bbef25dd89247f88fdfb052cf31840660889be
Jul 10 19:20:53.044227 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:53.044190    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" event={"ID":"c1ab6420-329d-4ad7-9529-c68ef65af108","Type":"ContainerStarted","Data":"df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040"}
Jul 10 19:20:53.044430 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:53.044246    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" event={"ID":"c1ab6420-329d-4ad7-9529-c68ef65af108","Type":"ContainerStarted","Data":"460fb597e85143805e711581d6bbef25dd89247f88fdfb052cf31840660889be"}
Jul 10 19:20:58.066541 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:58.066506    2567 generic.go:356] "Generic (PLEG): container finished" podID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerID="df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040" exitCode=0
Jul 10 19:20:58.066890 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:58.066568    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" event={"ID":"c1ab6420-329d-4ad7-9529-c68ef65af108","Type":"ContainerDied","Data":"df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040"}
Jul 10 19:20:59.072998 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:59.072958    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" event={"ID":"c1ab6420-329d-4ad7-9529-c68ef65af108","Type":"ContainerStarted","Data":"31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0"}
Jul 10 19:20:59.073480 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:59.073038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" event={"ID":"c1ab6420-329d-4ad7-9529-c68ef65af108","Type":"ContainerStarted","Data":"1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b"}
Jul 10 19:20:59.073480 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:59.073256    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:20:59.092464 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:20:59.092419    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" podStartSLOduration=1.9767691269999998 podStartE2EDuration="7.092405205s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.115636078s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:20:59.09104747 +0000 UTC m=+1362.342694799" watchObservedRunningTime="2026-07-10 19:20:59.092405205 +0000 UTC m=+1362.344052533"
Jul 10 19:21:00.076719 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:21:00.076686    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:21:06.087002 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:21:06.086973    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:22:36.418125 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:36.418044    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:22:42.484281 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:42.484246    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"]
Jul 10 19:22:42.484801 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:42.484496    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerName="kserve-container" containerID="cri-o://1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b" gracePeriod=30
Jul 10 19:22:42.484801 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:42.484540    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerName="kube-rbac-proxy" containerID="cri-o://31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0" gracePeriod=30
Jul 10 19:22:42.547992 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:42.547960    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:22:43.327213 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.327189    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:22:43.412009 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.411933    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1ab6420-329d-4ad7-9529-c68ef65af108-kserve-provision-location\") pod \"c1ab6420-329d-4ad7-9529-c68ef65af108\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") "
Jul 10 19:22:43.412009 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.411986    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-cabundle-cert\") pod \"c1ab6420-329d-4ad7-9529-c68ef65af108\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") "
Jul 10 19:22:43.412209 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.412014    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1ab6420-329d-4ad7-9529-c68ef65af108-proxy-tls\") pod \"c1ab6420-329d-4ad7-9529-c68ef65af108\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") "
Jul 10 19:22:43.412209 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.412061    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-rbac-proxy-sar-config\") pod \"c1ab6420-329d-4ad7-9529-c68ef65af108\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") "
Jul 10 19:22:43.412209 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.412092    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfq64\" (UniqueName: \"kubernetes.io/projected/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-api-access-cfq64\") pod \"c1ab6420-329d-4ad7-9529-c68ef65af108\" (UID: \"c1ab6420-329d-4ad7-9529-c68ef65af108\") "
Jul 10 19:22:43.412463 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.412429    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c1ab6420-329d-4ad7-9529-c68ef65af108" (UID: "c1ab6420-329d-4ad7-9529-c68ef65af108"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:22:43.412573 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.412451    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c1ab6420-329d-4ad7-9529-c68ef65af108" (UID: "c1ab6420-329d-4ad7-9529-c68ef65af108"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:22:43.414118 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.414092    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-api-access-cfq64" (OuterVolumeSpecName: "kube-api-access-cfq64") pod "c1ab6420-329d-4ad7-9529-c68ef65af108" (UID: "c1ab6420-329d-4ad7-9529-c68ef65af108"). InnerVolumeSpecName "kube-api-access-cfq64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:22:43.414200 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.414155    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ab6420-329d-4ad7-9529-c68ef65af108-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1ab6420-329d-4ad7-9529-c68ef65af108" (UID: "c1ab6420-329d-4ad7-9529-c68ef65af108"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:22:43.417565 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.417535    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ab6420-329d-4ad7-9529-c68ef65af108-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1ab6420-329d-4ad7-9529-c68ef65af108" (UID: "c1ab6420-329d-4ad7-9529-c68ef65af108"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:22:43.481738 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.481713    2567 generic.go:356] "Generic (PLEG): container finished" podID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerID="31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0" exitCode=2
Jul 10 19:22:43.481738 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.481736    2567 generic.go:356] "Generic (PLEG): container finished" podID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerID="1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b" exitCode=0
Jul 10 19:22:43.481897 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.481756    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" event={"ID":"c1ab6420-329d-4ad7-9529-c68ef65af108","Type":"ContainerDied","Data":"31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0"}
Jul 10 19:22:43.481897 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.481775    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"
Jul 10 19:22:43.481897 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.481799    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" event={"ID":"c1ab6420-329d-4ad7-9529-c68ef65af108","Type":"ContainerDied","Data":"1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b"}
Jul 10 19:22:43.481897 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.481820    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt" event={"ID":"c1ab6420-329d-4ad7-9529-c68ef65af108","Type":"ContainerDied","Data":"460fb597e85143805e711581d6bbef25dd89247f88fdfb052cf31840660889be"}
Jul 10 19:22:43.481897 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.481845    2567 scope.go:117] "RemoveContainer" containerID="31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0"
Jul 10 19:22:43.492895 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.491451    2567 scope.go:117] "RemoveContainer" containerID="1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b"
Jul 10 19:22:43.499935 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.499897    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"]
Jul 10 19:22:43.500145 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.500129    2567 scope.go:117] "RemoveContainer" containerID="df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040"
Jul 10 19:22:43.506584 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.506561    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-nc9vt"]
Jul 10 19:22:43.507715 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.507698    2567 scope.go:117] "RemoveContainer" containerID="31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0"
Jul 10 19:22:43.507976 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:22:43.507955    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0\": container with ID starting with 31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0 not found: ID does not exist" containerID="31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0"
Jul 10 19:22:43.508068 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.507986    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0"} err="failed to get container status \"31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0\": rpc error: code = NotFound desc = could not find container \"31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0\": container with ID starting with 31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0 not found: ID does not exist"
Jul 10 19:22:43.508068 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.508009    2567 scope.go:117] "RemoveContainer" containerID="1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b"
Jul 10 19:22:43.508277 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:22:43.508261    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b\": container with ID starting with 1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b not found: ID does not exist" containerID="1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b"
Jul 10 19:22:43.508316 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.508282    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b"} err="failed to get container status \"1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b\": rpc error: code = NotFound desc = could not find container \"1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b\": container with ID starting with 1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b not found: ID does not exist"
Jul 10 19:22:43.508316 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.508297    2567 scope.go:117] "RemoveContainer" containerID="df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040"
Jul 10 19:22:43.508503 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:22:43.508486    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040\": container with ID starting with df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040 not found: ID does not exist" containerID="df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040"
Jul 10 19:22:43.508566 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.508509    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040"} err="failed to get container status \"df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040\": rpc error: code = NotFound desc = could not find container \"df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040\": container with ID starting with df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040 not found: ID does not exist"
Jul 10 19:22:43.508566 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.508528    2567 scope.go:117] "RemoveContainer" containerID="31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0"
Jul 10 19:22:43.508746 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.508729    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0"} err="failed to get container status \"31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0\": rpc error: code = NotFound desc = could not find container \"31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0\": container with ID starting with 31b00d6880b2d30a09efbe6be50b6c27b81771a949c305c6c4e5ccadb64663d0 not found: ID does not exist"
Jul 10 19:22:43.508789 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.508748    2567 scope.go:117] "RemoveContainer" containerID="1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b"
Jul 10 19:22:43.508998 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.508981    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b"} err="failed to get container status \"1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b\": rpc error: code = NotFound desc = could not find container \"1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b\": container with ID starting with 1e448cd6e60d2012ae45103e17536cecf79e3b415f7a9b5ed35d8041f1ba630b not found: ID does not exist"
Jul 10 19:22:43.509046 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.508999    2567 scope.go:117] "RemoveContainer" containerID="df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040"
Jul 10 19:22:43.509193 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.509175    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040"} err="failed to get container status \"df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040\": rpc error: code = NotFound desc = could not find container \"df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040\": container with ID starting with df19b32feea985928ce81bbe95e9a46b1028f7f374e433c6e4bb56cd9b7a3040 not found: ID does not exist"
Jul 10 19:22:43.513555 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.513541    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-cabundle-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:22:43.513605 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.513559    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1ab6420-329d-4ad7-9529-c68ef65af108-proxy-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:22:43.513605 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.513568    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:22:43.513605 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.513577    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfq64\" (UniqueName: \"kubernetes.io/projected/c1ab6420-329d-4ad7-9529-c68ef65af108-kube-api-access-cfq64\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:22:43.513605 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:43.513585    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1ab6420-329d-4ad7-9529-c68ef65af108-kserve-provision-location\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:22:45.418266 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:45.418233    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" path="/var/lib/kubelet/pods/c1ab6420-329d-4ad7-9529-c68ef65af108/volumes"
Jul 10 19:22:47.668321 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.668293    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:22:47.722648 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.722612    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"]
Jul 10 19:22:47.723066 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.723051    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerName="kube-rbac-proxy"
Jul 10 19:22:47.723115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.723069    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerName="kube-rbac-proxy"
Jul 10 19:22:47.723115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.723089    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerName="storage-initializer"
Jul 10 19:22:47.723115 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.723098    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerName="storage-initializer"
Jul 10 19:22:47.723205 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.723127    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerName="kserve-container"
Jul 10 19:22:47.723205 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.723135    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerName="kserve-container"
Jul 10 19:22:47.723205 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.723194    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerName="kube-rbac-proxy"
Jul 10 19:22:47.723306 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.723205    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1ab6420-329d-4ad7-9529-c68ef65af108" containerName="kserve-container"
Jul 10 19:22:47.726489 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.726470    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.729226 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.729194    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 10 19:22:47.729341 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.729245    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 19:22:47.729341 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.729253    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:22:47.729466 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.729370    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 10 19:22:47.730206 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.730189    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 10 19:22:47.730294 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.730177    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z7zjv\""
Jul 10 19:22:47.730294 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.730196    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 19:22:47.735447 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.735423    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"]
Jul 10 19:22:47.849952 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.849924    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.850085 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.849962    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.850085 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.850007    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/008acf47-7844-41e9-a563-528c4836936b-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.850165 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.850086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh9zc\" (UniqueName: \"kubernetes.io/projected/008acf47-7844-41e9-a563-528c4836936b-kube-api-access-kh9zc\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.850165 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.850131    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/008acf47-7844-41e9-a563-528c4836936b-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.924190 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.924122    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:22:47.951222 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.951193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/008acf47-7844-41e9-a563-528c4836936b-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.951347 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.951254    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh9zc\" (UniqueName: \"kubernetes.io/projected/008acf47-7844-41e9-a563-528c4836936b-kube-api-access-kh9zc\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.951347 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.951298    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/008acf47-7844-41e9-a563-528c4836936b-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.951465 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.951345    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.951465 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.951384    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.951561 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.951538    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/008acf47-7844-41e9-a563-528c4836936b-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.952056 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.952033    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.952056 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.952046    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.953689 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.953671    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/008acf47-7844-41e9-a563-528c4836936b-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:47.961158 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:47.961126    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh9zc\" (UniqueName: \"kubernetes.io/projected/008acf47-7844-41e9-a563-528c4836936b-kube-api-access-kh9zc\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:48.038124 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:48.038101    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:22:48.166426 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:48.166403    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"]
Jul 10 19:22:48.168365 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:22:48.168334    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008acf47_7844_41e9_a563_528c4836936b.slice/crio-59748417084c1f1155966bfd4575a525484dc3a643d5e69234b0a018e7fd0f24 WatchSource:0}: Error finding container 59748417084c1f1155966bfd4575a525484dc3a643d5e69234b0a018e7fd0f24: Status 404 returned error can't find the container with id 59748417084c1f1155966bfd4575a525484dc3a643d5e69234b0a018e7fd0f24
Jul 10 19:22:48.504992 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:48.504893    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" event={"ID":"008acf47-7844-41e9-a563-528c4836936b","Type":"ContainerStarted","Data":"c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce"}
Jul 10 19:22:48.504992 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:22:48.504960    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" event={"ID":"008acf47-7844-41e9-a563-528c4836936b","Type":"ContainerStarted","Data":"59748417084c1f1155966bfd4575a525484dc3a643d5e69234b0a018e7fd0f24"}
Jul 10 19:23:22.647158 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:23:22.647121    2567 generic.go:356] "Generic (PLEG): container finished" podID="008acf47-7844-41e9-a563-528c4836936b" containerID="c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce" exitCode=0
Jul 10 19:23:22.647563 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:23:22.647194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" event={"ID":"008acf47-7844-41e9-a563-528c4836936b","Type":"ContainerDied","Data":"c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce"}
Jul 10 19:23:23.654598 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:23:23.654561    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" event={"ID":"008acf47-7844-41e9-a563-528c4836936b","Type":"ContainerStarted","Data":"0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd"}
Jul 10 19:23:23.655020 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:23:23.654620    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" event={"ID":"008acf47-7844-41e9-a563-528c4836936b","Type":"ContainerStarted","Data":"b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe"}
Jul 10 19:23:23.655020 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:23:23.654811    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:23:23.675153 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:23:23.675102    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" podStartSLOduration=2.80829141 podStartE2EDuration="36.675087181s" totalImagesPullingTime="0s" totalInitContainerRuntime="33.866795771s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:23:23.673577539 +0000 UTC m=+1506.925224882" watchObservedRunningTime="2026-07-10 19:23:23.675087181 +0000 UTC m=+1506.926734510"
Jul 10 19:23:24.659406 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:23:24.659375    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:23:30.668402 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:23:30.668375    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:24:52.417306 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:52.417273    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:24:57.796529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:57.796498    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"]
Jul 10 19:24:57.797008 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:57.796731    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" podUID="008acf47-7844-41e9-a563-528c4836936b" containerName="kserve-container" containerID="cri-o://b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe" gracePeriod=30
Jul 10 19:24:57.797008 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:57.796806    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" podUID="008acf47-7844-41e9-a563-528c4836936b" containerName="kube-rbac-proxy" containerID="cri-o://0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd" gracePeriod=30
Jul 10 19:24:58.030014 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.029973    2567 generic.go:356] "Generic (PLEG): container finished" podID="008acf47-7844-41e9-a563-528c4836936b" containerID="0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd" exitCode=2
Jul 10 19:24:58.030191 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.030045    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" event={"ID":"008acf47-7844-41e9-a563-528c4836936b","Type":"ContainerDied","Data":"0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd"}
Jul 10 19:24:58.504667 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.504620    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:24:58.643469 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.643444    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:24:58.771037 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.770967    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-kube-rbac-proxy-sar-config\") pod \"008acf47-7844-41e9-a563-528c4836936b\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") "
Jul 10 19:24:58.771037 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.771003    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh9zc\" (UniqueName: \"kubernetes.io/projected/008acf47-7844-41e9-a563-528c4836936b-kube-api-access-kh9zc\") pod \"008acf47-7844-41e9-a563-528c4836936b\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") "
Jul 10 19:24:58.771037 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.771034    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-cabundle-cert\") pod \"008acf47-7844-41e9-a563-528c4836936b\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") "
Jul 10 19:24:58.771292 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.771051    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/008acf47-7844-41e9-a563-528c4836936b-proxy-tls\") pod \"008acf47-7844-41e9-a563-528c4836936b\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") "
Jul 10 19:24:58.771292 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.771100    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/008acf47-7844-41e9-a563-528c4836936b-kserve-provision-location\") pod \"008acf47-7844-41e9-a563-528c4836936b\" (UID: \"008acf47-7844-41e9-a563-528c4836936b\") "
Jul 10 19:24:58.771511 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.771390    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "008acf47-7844-41e9-a563-528c4836936b" (UID: "008acf47-7844-41e9-a563-528c4836936b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:24:58.771613 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.771392    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "008acf47-7844-41e9-a563-528c4836936b" (UID: "008acf47-7844-41e9-a563-528c4836936b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:24:58.773069 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.773041    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008acf47-7844-41e9-a563-528c4836936b-kube-api-access-kh9zc" (OuterVolumeSpecName: "kube-api-access-kh9zc") pod "008acf47-7844-41e9-a563-528c4836936b" (UID: "008acf47-7844-41e9-a563-528c4836936b"). InnerVolumeSpecName "kube-api-access-kh9zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:24:58.773069 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.773042    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008acf47-7844-41e9-a563-528c4836936b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "008acf47-7844-41e9-a563-528c4836936b" (UID: "008acf47-7844-41e9-a563-528c4836936b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:24:58.777376 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.777345    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008acf47-7844-41e9-a563-528c4836936b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "008acf47-7844-41e9-a563-528c4836936b" (UID: "008acf47-7844-41e9-a563-528c4836936b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:24:58.872243 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.872206    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-cabundle-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:24:58.872243 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.872243    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/008acf47-7844-41e9-a563-528c4836936b-proxy-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:24:58.872243 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.872253    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/008acf47-7844-41e9-a563-528c4836936b-kserve-provision-location\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:24:58.872629 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.872264    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/008acf47-7844-41e9-a563-528c4836936b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:24:58.872629 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:58.872272    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh9zc\" (UniqueName: \"kubernetes.io/projected/008acf47-7844-41e9-a563-528c4836936b-kube-api-access-kh9zc\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:24:59.036221 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.036149    2567 generic.go:356] "Generic (PLEG): container finished" podID="008acf47-7844-41e9-a563-528c4836936b" containerID="b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe" exitCode=0
Jul 10 19:24:59.036221 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.036212    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"
Jul 10 19:24:59.036365 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.036230    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" event={"ID":"008acf47-7844-41e9-a563-528c4836936b","Type":"ContainerDied","Data":"b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe"}
Jul 10 19:24:59.036365 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.036300    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5" event={"ID":"008acf47-7844-41e9-a563-528c4836936b","Type":"ContainerDied","Data":"59748417084c1f1155966bfd4575a525484dc3a643d5e69234b0a018e7fd0f24"}
Jul 10 19:24:59.036365 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.036327    2567 scope.go:117] "RemoveContainer" containerID="0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd"
Jul 10 19:24:59.045155 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.045134    2567 scope.go:117] "RemoveContainer" containerID="b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe"
Jul 10 19:24:59.052541 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.052523    2567 scope.go:117] "RemoveContainer" containerID="c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce"
Jul 10 19:24:59.058081 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.058061    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"]
Jul 10 19:24:59.059667 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.059654    2567 scope.go:117] "RemoveContainer" containerID="0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd"
Jul 10 19:24:59.059896 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:24:59.059872    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd\": container with ID starting with 0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd not found: ID does not exist" containerID="0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd"
Jul 10 19:24:59.059987 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.059927    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd"} err="failed to get container status \"0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd\": rpc error: code = NotFound desc = could not find container \"0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd\": container with ID starting with 0ab38caaffc160ebb85e7636a400a935f6673590a5197e7ae834d22f634631bd not found: ID does not exist"
Jul 10 19:24:59.059987 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.059944    2567 scope.go:117] "RemoveContainer" containerID="b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe"
Jul 10 19:24:59.060300 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:24:59.060275    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe\": container with ID starting with b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe not found: ID does not exist" containerID="b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe"
Jul 10 19:24:59.060365 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.060308    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe"} err="failed to get container status \"b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe\": rpc error: code = NotFound desc = could not find container \"b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe\": container with ID starting with b5cbb6772dddba95bd19354febddbdc9eace2e4e249de343da5fd0e13f820bbe not found: ID does not exist"
Jul 10 19:24:59.060365 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.060332    2567 scope.go:117] "RemoveContainer" containerID="c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce"
Jul 10 19:24:59.060682 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:24:59.060659    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce\": container with ID starting with c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce not found: ID does not exist" containerID="c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce"
Jul 10 19:24:59.061003 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.060690    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce"} err="failed to get container status \"c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce\": rpc error: code = NotFound desc = could not find container \"c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce\": container with ID starting with c9c86e3012e27a492db9d9e8fc002ec4b274bcb33e6226101ec767609da6a7ce not found: ID does not exist"
Jul 10 19:24:59.062396 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.062368    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sqnd5"]
Jul 10 19:24:59.417985 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:24:59.417951    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="008acf47-7844-41e9-a563-528c4836936b" path="/var/lib/kubelet/pods/008acf47-7844-41e9-a563-528c4836936b/volumes"
Jul 10 19:25:02.947446 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:02.947411    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:25:03.007956 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.007900    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"]
Jul 10 19:25:03.008317 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.008292    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="008acf47-7844-41e9-a563-528c4836936b" containerName="kserve-container"
Jul 10 19:25:03.008317 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.008320    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="008acf47-7844-41e9-a563-528c4836936b" containerName="kserve-container"
Jul 10 19:25:03.008490 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.008334    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="008acf47-7844-41e9-a563-528c4836936b" containerName="kube-rbac-proxy"
Jul 10 19:25:03.008490 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.008342    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="008acf47-7844-41e9-a563-528c4836936b" containerName="kube-rbac-proxy"
Jul 10 19:25:03.008490 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.008363    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="008acf47-7844-41e9-a563-528c4836936b" containerName="storage-initializer"
Jul 10 19:25:03.008490 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.008368    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="008acf47-7844-41e9-a563-528c4836936b" containerName="storage-initializer"
Jul 10 19:25:03.008490 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.008435    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="008acf47-7844-41e9-a563-528c4836936b" containerName="kube-rbac-proxy"
Jul 10 19:25:03.008490 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.008447    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="008acf47-7844-41e9-a563-528c4836936b" containerName="kserve-container"
Jul 10 19:25:03.013550 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.013532    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.019455 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.019430    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 10 19:25:03.019601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.019475    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 10 19:25:03.019601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.019487    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 19:25:03.019601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.019556    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:25:03.020122 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.019754    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 19:25:03.020733 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.020713    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 10 19:25:03.021001 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.020763    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z7zjv\""
Jul 10 19:25:03.022687 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.022661    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"]
Jul 10 19:25:03.041279 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.041254    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:25:03.105278 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.105245    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z94g6\" (UniqueName: \"kubernetes.io/projected/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-api-access-z94g6\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.105460 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.105332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.105460 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.105355    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.105460 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.105421    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.105460 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.105439    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.206157 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.206067    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.206157 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.206112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.206157 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.206152    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.206413 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.206169    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.206413 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.206190    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z94g6\" (UniqueName: \"kubernetes.io/projected/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-api-access-z94g6\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.206650 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.206629    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.206829 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.206810    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.206964 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.206899    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.208493 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.208475    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.214742 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.214719    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z94g6\" (UniqueName: \"kubernetes.io/projected/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-api-access-z94g6\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.324698 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.324659    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:03.460875 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:03.460851    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"]
Jul 10 19:25:03.463070 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:25:03.463037    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766495ed_b1c2_4ec6_b1f2_17ac15d31a08.slice/crio-3537950c04057ba3c720263d76cb04cf6e51502e3236fb31f5c996bdd00b55be WatchSource:0}: Error finding container 3537950c04057ba3c720263d76cb04cf6e51502e3236fb31f5c996bdd00b55be: Status 404 returned error can't find the container with id 3537950c04057ba3c720263d76cb04cf6e51502e3236fb31f5c996bdd00b55be
Jul 10 19:25:04.061303 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:04.061264    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" event={"ID":"766495ed-b1c2-4ec6-b1f2-17ac15d31a08","Type":"ContainerStarted","Data":"97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405"}
Jul 10 19:25:04.061656 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:04.061337    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" event={"ID":"766495ed-b1c2-4ec6-b1f2-17ac15d31a08","Type":"ContainerStarted","Data":"3537950c04057ba3c720263d76cb04cf6e51502e3236fb31f5c996bdd00b55be"}
Jul 10 19:25:14.106089 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:14.106057    2567 generic.go:356] "Generic (PLEG): container finished" podID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerID="97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405" exitCode=0
Jul 10 19:25:14.106447 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:14.106111    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" event={"ID":"766495ed-b1c2-4ec6-b1f2-17ac15d31a08","Type":"ContainerDied","Data":"97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405"}
Jul 10 19:25:15.113409 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:15.113378    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" event={"ID":"766495ed-b1c2-4ec6-b1f2-17ac15d31a08","Type":"ContainerStarted","Data":"2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5"}
Jul 10 19:25:15.113828 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:15.113439    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" event={"ID":"766495ed-b1c2-4ec6-b1f2-17ac15d31a08","Type":"ContainerStarted","Data":"6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710"}
Jul 10 19:25:15.113828 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:15.113642    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:15.135053 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:15.135005    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" podStartSLOduration=3.060695536 podStartE2EDuration="13.134989753s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.074294217s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:15.133486318 +0000 UTC m=+1618.385133663" watchObservedRunningTime="2026-07-10 19:25:15.134989753 +0000 UTC m=+1618.386637082"
Jul 10 19:25:16.118100 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:16.118063    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:16.119315 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:16.119283    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:25:17.123104 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:17.123069    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:25:22.127691 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:22.127660    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:22.128261 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:22.128233    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:25:32.129060 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:32.129030    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:33.045894 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.045858    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"]
Jul 10 19:25:33.046131 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.046105    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kserve-container" containerID="cri-o://6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710" gracePeriod=30
Jul 10 19:25:33.046195 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.046155    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kube-rbac-proxy" containerID="cri-o://2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5" gracePeriod=30
Jul 10 19:25:33.074017 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.073980    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:25:33.189444 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.189415    2567 generic.go:356] "Generic (PLEG): container finished" podID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerID="2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5" exitCode=2
Jul 10 19:25:33.189765 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.189449    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" event={"ID":"766495ed-b1c2-4ec6-b1f2-17ac15d31a08","Type":"ContainerDied","Data":"2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5"}
Jul 10 19:25:33.788867 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.788844    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:33.845853 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.845830    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-proxy-tls\") pod \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") "
Jul 10 19:25:33.846007 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.845866    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-cabundle-cert\") pod \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") "
Jul 10 19:25:33.846007 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.845886    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kserve-provision-location\") pod \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") "
Jul 10 19:25:33.846007 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.845935    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z94g6\" (UniqueName: \"kubernetes.io/projected/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-api-access-z94g6\") pod \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") "
Jul 10 19:25:33.846183 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.846019    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-rbac-proxy-sar-config\") pod \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\" (UID: \"766495ed-b1c2-4ec6-b1f2-17ac15d31a08\") "
Jul 10 19:25:33.846330 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.846295    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "766495ed-b1c2-4ec6-b1f2-17ac15d31a08" (UID: "766495ed-b1c2-4ec6-b1f2-17ac15d31a08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:25:33.846415 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.846347    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "766495ed-b1c2-4ec6-b1f2-17ac15d31a08" (UID: "766495ed-b1c2-4ec6-b1f2-17ac15d31a08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:25:33.846853 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.846834    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "766495ed-b1c2-4ec6-b1f2-17ac15d31a08" (UID: "766495ed-b1c2-4ec6-b1f2-17ac15d31a08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:25:33.848018 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.847992    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-api-access-z94g6" (OuterVolumeSpecName: "kube-api-access-z94g6") pod "766495ed-b1c2-4ec6-b1f2-17ac15d31a08" (UID: "766495ed-b1c2-4ec6-b1f2-17ac15d31a08"). InnerVolumeSpecName "kube-api-access-z94g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:25:33.848073 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.848031    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "766495ed-b1c2-4ec6-b1f2-17ac15d31a08" (UID: "766495ed-b1c2-4ec6-b1f2-17ac15d31a08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:25:33.947620 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.947544    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-proxy-tls\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:25:33.947620 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.947571    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-cabundle-cert\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:25:33.947620 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.947585    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kserve-provision-location\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:25:33.947620 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.947597    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z94g6\" (UniqueName: \"kubernetes.io/projected/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-api-access-z94g6\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:25:33.947620 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:33.947609    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/766495ed-b1c2-4ec6-b1f2-17ac15d31a08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:25:34.195248 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.195213    2567 generic.go:356] "Generic (PLEG): container finished" podID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerID="6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710" exitCode=0
Jul 10 19:25:34.195601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.195262    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" event={"ID":"766495ed-b1c2-4ec6-b1f2-17ac15d31a08","Type":"ContainerDied","Data":"6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710"}
Jul 10 19:25:34.195601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.195284    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"
Jul 10 19:25:34.195601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.195334    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s" event={"ID":"766495ed-b1c2-4ec6-b1f2-17ac15d31a08","Type":"ContainerDied","Data":"3537950c04057ba3c720263d76cb04cf6e51502e3236fb31f5c996bdd00b55be"}
Jul 10 19:25:34.195601 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.195371    2567 scope.go:117] "RemoveContainer" containerID="2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5"
Jul 10 19:25:34.208021 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.207997    2567 scope.go:117] "RemoveContainer" containerID="6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710"
Jul 10 19:25:34.215288 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.215266    2567 scope.go:117] "RemoveContainer" containerID="97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405"
Jul 10 19:25:34.221087 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.221063    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"]
Jul 10 19:25:34.222340 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.222323    2567 scope.go:117] "RemoveContainer" containerID="2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5"
Jul 10 19:25:34.222600 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:25:34.222579    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5\": container with ID starting with 2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5 not found: ID does not exist" containerID="2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5"
Jul 10 19:25:34.222674 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.222612    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5"} err="failed to get container status \"2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5\": rpc error: code = NotFound desc = could not find container \"2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5\": container with ID starting with 2ddbf23f924e172ef81c7d3360381755ebcabac0710e970fb34e8c1989e944e5 not found: ID does not exist"
Jul 10 19:25:34.222674 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.222636    2567 scope.go:117] "RemoveContainer" containerID="6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710"
Jul 10 19:25:34.222851 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:25:34.222833    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710\": container with ID starting with 6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710 not found: ID does not exist" containerID="6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710"
Jul 10 19:25:34.222897 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.222856    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710"} err="failed to get container status \"6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710\": rpc error: code = NotFound desc = could not find container \"6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710\": container with ID starting with 6ff59092b89934eaeed6977af5e7a9dc57d35af9aa82ed2099dd87177d474710 not found: ID does not exist"
Jul 10 19:25:34.222897 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.222871    2567 scope.go:117] "RemoveContainer" containerID="97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405"
Jul 10 19:25:34.223134 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:25:34.223117    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405\": container with ID starting with 97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405 not found: ID does not exist" containerID="97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405"
Jul 10 19:25:34.223179 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.223138    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405"} err="failed to get container status \"97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405\": rpc error: code = NotFound desc = could not find container \"97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405\": container with ID starting with 97e6735dcf6109ae30466c3f2b2924c4f6af0448470d82b6726f782d09329405 not found: ID does not exist"
Jul 10 19:25:34.225700 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:34.225681    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-8mg7s"]
Jul 10 19:25:35.418540 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:35.418510    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" path="/var/lib/kubelet/pods/766495ed-b1c2-4ec6-b1f2-17ac15d31a08/volumes"
Jul 10 19:25:39.704848 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.704816    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qt25s/must-gather-bxtr6"]
Jul 10 19:25:39.705307 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.705131    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kserve-container"
Jul 10 19:25:39.705307 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.705143    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kserve-container"
Jul 10 19:25:39.705307 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.705158    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="storage-initializer"
Jul 10 19:25:39.705307 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.705166    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="storage-initializer"
Jul 10 19:25:39.705307 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.705175    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kube-rbac-proxy"
Jul 10 19:25:39.705307 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.705180    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kube-rbac-proxy"
Jul 10 19:25:39.705307 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.705239    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kube-rbac-proxy"
Jul 10 19:25:39.705307 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.705250    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="766495ed-b1c2-4ec6-b1f2-17ac15d31a08" containerName="kserve-container"
Jul 10 19:25:39.709831 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.709811    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:25:39.712220 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.712198    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qt25s\"/\"kube-root-ca.crt\""
Jul 10 19:25:39.713149 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.713123    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qt25s\"/\"default-dockercfg-5lm2f\""
Jul 10 19:25:39.713247 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.713173    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qt25s\"/\"openshift-service-ca.crt\""
Jul 10 19:25:39.720322 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.720303    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qt25s/must-gather-bxtr6"]
Jul 10 19:25:39.723352 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.723330    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:25:39.789012 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.788989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:25:39.791069 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.791047    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/197beb33-0a14-4306-a989-de5d5c3cf07c-must-gather-output\") pod \"must-gather-bxtr6\" (UID: \"197beb33-0a14-4306-a989-de5d5c3cf07c\") " pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:25:39.791153 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.791086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5sg\" (UniqueName: \"kubernetes.io/projected/197beb33-0a14-4306-a989-de5d5c3cf07c-kube-api-access-zb5sg\") pod \"must-gather-bxtr6\" (UID: \"197beb33-0a14-4306-a989-de5d5c3cf07c\") " pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:25:39.892053 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.892027    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/197beb33-0a14-4306-a989-de5d5c3cf07c-must-gather-output\") pod \"must-gather-bxtr6\" (UID: \"197beb33-0a14-4306-a989-de5d5c3cf07c\") " pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:25:39.892151 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.892060    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5sg\" (UniqueName: \"kubernetes.io/projected/197beb33-0a14-4306-a989-de5d5c3cf07c-kube-api-access-zb5sg\") pod \"must-gather-bxtr6\" (UID: \"197beb33-0a14-4306-a989-de5d5c3cf07c\") " pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:25:39.892366 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.892344    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/197beb33-0a14-4306-a989-de5d5c3cf07c-must-gather-output\") pod \"must-gather-bxtr6\" (UID: \"197beb33-0a14-4306-a989-de5d5c3cf07c\") " pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:25:39.899632 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:39.899610    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5sg\" (UniqueName: \"kubernetes.io/projected/197beb33-0a14-4306-a989-de5d5c3cf07c-kube-api-access-zb5sg\") pod \"must-gather-bxtr6\" (UID: \"197beb33-0a14-4306-a989-de5d5c3cf07c\") " pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:25:40.032868 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:40.032781    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:25:40.160060 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:40.160036    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qt25s/must-gather-bxtr6"]
Jul 10 19:25:40.161981 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:25:40.161954    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod197beb33_0a14_4306_a989_de5d5c3cf07c.slice/crio-e1d5ad88e0c14ada7b09608017152f312a58759304242a25532898f16ea97f1f WatchSource:0}: Error finding container e1d5ad88e0c14ada7b09608017152f312a58759304242a25532898f16ea97f1f: Status 404 returned error can't find the container with id e1d5ad88e0c14ada7b09608017152f312a58759304242a25532898f16ea97f1f
Jul 10 19:25:40.163781 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:40.163761    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:25:40.219167 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:40.219141    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qt25s/must-gather-bxtr6" event={"ID":"197beb33-0a14-4306-a989-de5d5c3cf07c","Type":"ContainerStarted","Data":"e1d5ad88e0c14ada7b09608017152f312a58759304242a25532898f16ea97f1f"}
Jul 10 19:25:46.249166 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:46.249133    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qt25s/must-gather-bxtr6" event={"ID":"197beb33-0a14-4306-a989-de5d5c3cf07c","Type":"ContainerStarted","Data":"e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e"}
Jul 10 19:25:47.255341 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:47.255297    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qt25s/must-gather-bxtr6" event={"ID":"197beb33-0a14-4306-a989-de5d5c3cf07c","Type":"ContainerStarted","Data":"766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416"}
Jul 10 19:25:47.271851 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:25:47.271795    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qt25s/must-gather-bxtr6" podStartSLOduration=2.332477764 podStartE2EDuration="8.271776515s" totalImagesPullingTime="5.939298751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:47.271372284 +0000 UTC m=+1650.523019625" watchObservedRunningTime="2026-07-10 19:25:47.271776515 +0000 UTC m=+1650.523423845"
Jul 10 19:26:06.344438 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:06.344403    2567 generic.go:356] "Generic (PLEG): container finished" podID="197beb33-0a14-4306-a989-de5d5c3cf07c" containerID="e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e" exitCode=0
Jul 10 19:26:06.344994 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:06.344482    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qt25s/must-gather-bxtr6" event={"ID":"197beb33-0a14-4306-a989-de5d5c3cf07c","Type":"ContainerDied","Data":"e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e"}
Jul 10 19:26:06.344994 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:06.344790    2567 scope.go:117] "RemoveContainer" containerID="e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e"
Jul 10 19:26:06.981057 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:06.981030    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qt25s_must-gather-bxtr6_197beb33-0a14-4306-a989-de5d5c3cf07c/gather/0.log"
Jul 10 19:26:07.389313 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.389287    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:26:07.516317 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.516291    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-znhzn/must-gather-4nd47"]
Jul 10 19:26:07.518577 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.518561    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-znhzn/must-gather-4nd47"
Jul 10 19:26:07.521002 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.520968    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-znhzn\"/\"openshift-service-ca.crt\""
Jul 10 19:26:07.521148 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.521098    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-znhzn\"/\"default-dockercfg-h4thm\""
Jul 10 19:26:07.521148 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.521140    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-znhzn\"/\"kube-root-ca.crt\""
Jul 10 19:26:07.528373 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.528355    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-znhzn/must-gather-4nd47"]
Jul 10 19:26:07.646572 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.646501    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296fw\" (UniqueName: \"kubernetes.io/projected/839ac5aa-04f9-4d16-a037-42c09634fa68-kube-api-access-296fw\") pod \"must-gather-4nd47\" (UID: \"839ac5aa-04f9-4d16-a037-42c09634fa68\") " pod="openshift-must-gather-znhzn/must-gather-4nd47"
Jul 10 19:26:07.646572 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.646545    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/839ac5aa-04f9-4d16-a037-42c09634fa68-must-gather-output\") pod \"must-gather-4nd47\" (UID: \"839ac5aa-04f9-4d16-a037-42c09634fa68\") " pod="openshift-must-gather-znhzn/must-gather-4nd47"
Jul 10 19:26:07.747250 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.747204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296fw\" (UniqueName: \"kubernetes.io/projected/839ac5aa-04f9-4d16-a037-42c09634fa68-kube-api-access-296fw\") pod \"must-gather-4nd47\" (UID: \"839ac5aa-04f9-4d16-a037-42c09634fa68\") " pod="openshift-must-gather-znhzn/must-gather-4nd47"
Jul 10 19:26:07.747399 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.747306    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/839ac5aa-04f9-4d16-a037-42c09634fa68-must-gather-output\") pod \"must-gather-4nd47\" (UID: \"839ac5aa-04f9-4d16-a037-42c09634fa68\") " pod="openshift-must-gather-znhzn/must-gather-4nd47"
Jul 10 19:26:07.747782 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.747766    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/839ac5aa-04f9-4d16-a037-42c09634fa68-must-gather-output\") pod \"must-gather-4nd47\" (UID: \"839ac5aa-04f9-4d16-a037-42c09634fa68\") " pod="openshift-must-gather-znhzn/must-gather-4nd47"
Jul 10 19:26:07.755834 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.755805    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-296fw\" (UniqueName: \"kubernetes.io/projected/839ac5aa-04f9-4d16-a037-42c09634fa68-kube-api-access-296fw\") pod \"must-gather-4nd47\" (UID: \"839ac5aa-04f9-4d16-a037-42c09634fa68\") " pod="openshift-must-gather-znhzn/must-gather-4nd47"
Jul 10 19:26:07.827428 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.827395    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-znhzn/must-gather-4nd47"
Jul 10 19:26:07.950529 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:07.950503    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-znhzn/must-gather-4nd47"]
Jul 10 19:26:07.952018 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:26:07.951991    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod839ac5aa_04f9_4d16_a037_42c09634fa68.slice/crio-e3c65dc7da694218e2c356432bfe76d20c83c03bf5735376e9e79cab34a470c4 WatchSource:0}: Error finding container e3c65dc7da694218e2c356432bfe76d20c83c03bf5735376e9e79cab34a470c4: Status 404 returned error can't find the container with id e3c65dc7da694218e2c356432bfe76d20c83c03bf5735376e9e79cab34a470c4
Jul 10 19:26:08.354499 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:08.354466    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znhzn/must-gather-4nd47" event={"ID":"839ac5aa-04f9-4d16-a037-42c09634fa68","Type":"ContainerStarted","Data":"e3c65dc7da694218e2c356432bfe76d20c83c03bf5735376e9e79cab34a470c4"}
Jul 10 19:26:08.643304 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:08.643217    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:26:09.360984 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:09.360945    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znhzn/must-gather-4nd47" event={"ID":"839ac5aa-04f9-4d16-a037-42c09634fa68","Type":"ContainerStarted","Data":"f254e9fe72c3c09f12122bcbb1f31fb6f22a1a898260c74f1e0538db9c4713a1"}
Jul 10 19:26:09.361179 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:09.360995    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znhzn/must-gather-4nd47" event={"ID":"839ac5aa-04f9-4d16-a037-42c09634fa68","Type":"ContainerStarted","Data":"3c3a1f131988935a8e87995766330e2be3c858391bf528028cd42fe7349dc40e"}
Jul 10 19:26:09.379350 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:09.379286    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-znhzn/must-gather-4nd47" podStartSLOduration=1.648709953 podStartE2EDuration="2.379257409s" totalImagesPullingTime="730.547456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:26:09.377244339 +0000 UTC m=+1672.628891670" watchObservedRunningTime="2026-07-10 19:26:09.379257409 +0000 UTC m=+1672.630904739"
Jul 10 19:26:10.106762 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:10.106731    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4xprr_d68b9eae-5b48-4dfa-bfd7-185a928fabd7/global-pull-secret-syncer/0.log"
Jul 10 19:26:10.258346 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:10.258315    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log"
Jul 10 19:26:10.369687 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:10.369619    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-15.ec2.internal_d18d246e0bedf5b391d12d0230022e47/haproxy/0.log"
Jul 10 19:26:12.369384 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.369342    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-qt25s/must-gather-bxtr6"]
Jul 10 19:26:12.379293 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.379258    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-qt25s/must-gather-bxtr6"]
Jul 10 19:26:12.380175 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.379639    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-qt25s/must-gather-bxtr6" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" containerName="copy" containerID="cri-o://766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416" gracePeriod=2
Jul 10 19:26:12.381926 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:26:12.381884    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bxtr6\" is forbidden: User \"system:node:ip-10-0-136-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-qt25s\": no relationship found between node 'ip-10-0-136-15.ec2.internal' and this object" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:26:12.739924 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.737522    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qt25s_must-gather-bxtr6_197beb33-0a14-4306-a989-de5d5c3cf07c/copy/0.log"
Jul 10 19:26:12.744405 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.741642    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:26:12.745995 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:26:12.745944    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bxtr6\" is forbidden: User \"system:node:ip-10-0-136-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-qt25s\": no relationship found between node 'ip-10-0-136-15.ec2.internal' and this object" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:26:12.793681 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.793640    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/197beb33-0a14-4306-a989-de5d5c3cf07c-must-gather-output\") pod \"197beb33-0a14-4306-a989-de5d5c3cf07c\" (UID: \"197beb33-0a14-4306-a989-de5d5c3cf07c\") "
Jul 10 19:26:12.796234 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.795130    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197beb33-0a14-4306-a989-de5d5c3cf07c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "197beb33-0a14-4306-a989-de5d5c3cf07c" (UID: "197beb33-0a14-4306-a989-de5d5c3cf07c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:26:12.796778 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.796721    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb5sg\" (UniqueName: \"kubernetes.io/projected/197beb33-0a14-4306-a989-de5d5c3cf07c-kube-api-access-zb5sg\") pod \"197beb33-0a14-4306-a989-de5d5c3cf07c\" (UID: \"197beb33-0a14-4306-a989-de5d5c3cf07c\") "
Jul 10 19:26:12.797338 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.797298    2567 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/197beb33-0a14-4306-a989-de5d5c3cf07c-must-gather-output\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:26:12.800366 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.800324    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197beb33-0a14-4306-a989-de5d5c3cf07c-kube-api-access-zb5sg" (OuterVolumeSpecName: "kube-api-access-zb5sg") pod "197beb33-0a14-4306-a989-de5d5c3cf07c" (UID: "197beb33-0a14-4306-a989-de5d5c3cf07c"). InnerVolumeSpecName "kube-api-access-zb5sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:26:12.898637 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:12.898351    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb5sg\" (UniqueName: \"kubernetes.io/projected/197beb33-0a14-4306-a989-de5d5c3cf07c-kube-api-access-zb5sg\") on node \"ip-10-0-136-15.ec2.internal\" DevicePath \"\""
Jul 10 19:26:13.085813 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.085701    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:26:13.194413 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.194319    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64zsj_7b8009c7-2984-43f5-8195-8c13750d2f2b/konnectivity-agent/0.log" limit=1024
Jul 10 19:26:13.397801 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.397718    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qt25s_must-gather-bxtr6_197beb33-0a14-4306-a989-de5d5c3cf07c/copy/0.log"
Jul 10 19:26:13.410490 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.410454    2567 generic.go:356] "Generic (PLEG): container finished" podID="197beb33-0a14-4306-a989-de5d5c3cf07c" containerID="766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416" exitCode=143
Jul 10 19:26:13.410661 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.410554    2567 scope.go:117] "RemoveContainer" containerID="766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416"
Jul 10 19:26:13.410782 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.410735    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:26:13.413200 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:26:13.413172    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bxtr6\" is forbidden: User \"system:node:ip-10-0-136-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-qt25s\": no relationship found between node 'ip-10-0-136-15.ec2.internal' and this object" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:26:13.438179 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.438138    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" path="/var/lib/kubelet/pods/197beb33-0a14-4306-a989-de5d5c3cf07c/volumes"
Jul 10 19:26:13.438939 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:26:13.438876    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bxtr6\" is forbidden: User \"system:node:ip-10-0-136-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-qt25s\": no relationship found between node 'ip-10-0-136-15.ec2.internal' and this object" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" pod="openshift-must-gather-qt25s/must-gather-bxtr6"
Jul 10 19:26:13.447620 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.447591    2567 scope.go:117] "RemoveContainer" containerID="e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e"
Jul 10 19:26:13.474481 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.474459    2567 scope.go:117] "RemoveContainer" containerID="766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416"
Jul 10 19:26:13.475052 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:26:13.474875    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416\": container with ID starting with 766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416 not found: ID does not exist" containerID="766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416"
Jul 10 19:26:13.475052 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.474927    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416"} err="failed to get container status \"766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416\": rpc error: code = NotFound desc = could not find container \"766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416\": container with ID starting with 766778eee9c3014117b488d8247997db12dd7ab7d5038c837e6bc0e74260e416 not found: ID does not exist"
Jul 10 19:26:13.475052 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.474955    2567 scope.go:117] "RemoveContainer" containerID="e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e"
Jul 10 19:26:13.475619 ip-10-0-136-15 kubenswrapper[2567]: E0710 19:26:13.475552    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e\": container with ID starting with e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e not found: ID does not exist" containerID="e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e"
Jul 10 19:26:13.475619 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:13.475587    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e"} err="failed to get container status \"e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e\": rpc error: code = NotFound desc = could not find container \"e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e\": container with ID starting with e4f4ed761a83df60d2d42ce8bc7af5ca98d6f7934e7ac617297f0f6b9dd86d5e not found: ID does not exist"
Jul 10 19:26:15.364606 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.364576    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxg79_61e594fb-2523-455e-b91d-5b9662e902e1/node-exporter/0.log"
Jul 10 19:26:15.387696 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.387661    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxg79_61e594fb-2523-455e-b91d-5b9662e902e1/kube-rbac-proxy/0.log"
Jul 10 19:26:15.415104 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.414975    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxg79_61e594fb-2523-455e-b91d-5b9662e902e1/init-textfile/0.log"
Jul 10 19:26:15.523384 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.523349    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4c04f04-d1ad-4e70-b592-9572ff12d48d/prometheus/0.log"
Jul 10 19:26:15.547645 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.547612    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4c04f04-d1ad-4e70-b592-9572ff12d48d/config-reloader/0.log"
Jul 10 19:26:15.573388 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.573364    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4c04f04-d1ad-4e70-b592-9572ff12d48d/thanos-sidecar/0.log"
Jul 10 19:26:15.596397 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.596365    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4c04f04-d1ad-4e70-b592-9572ff12d48d/kube-rbac-proxy-web/0.log"
Jul 10 19:26:15.616878 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.616828    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4c04f04-d1ad-4e70-b592-9572ff12d48d/kube-rbac-proxy/0.log"
Jul 10 19:26:15.639381 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.639301    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4c04f04-d1ad-4e70-b592-9572ff12d48d/kube-rbac-proxy-thanos/0.log"
Jul 10 19:26:15.661031 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.660970    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4c04f04-d1ad-4e70-b592-9572ff12d48d/init-config-reloader/0.log"
Jul 10 19:26:15.692697 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.692277    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-tm824_d6066461-2dd6-4e31-b512-7b1502f23a96/prometheus-operator/0.log"
Jul 10 19:26:15.711942 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.711899    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-tm824_d6066461-2dd6-4e31-b512-7b1502f23a96/kube-rbac-proxy/0.log"
Jul 10 19:26:15.776846 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.776811    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fd8dd5986-qw76j_5356aebe-d543-4fd8-a974-b92ec16ed048/telemeter-client/0.log"
Jul 10 19:26:15.801869 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.801835    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fd8dd5986-qw76j_5356aebe-d543-4fd8-a974-b92ec16ed048/reload/0.log"
Jul 10 19:26:15.848139 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.848106    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fd8dd5986-qw76j_5356aebe-d543-4fd8-a974-b92ec16ed048/kube-rbac-proxy/0.log"
Jul 10 19:26:15.889400 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.889362    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77c964dcc5-89zt7_b6c7a7f8-98f7-4469-a203-be4c8eb99869/thanos-query/0.log"
Jul 10 19:26:15.914521 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.914487    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77c964dcc5-89zt7_b6c7a7f8-98f7-4469-a203-be4c8eb99869/kube-rbac-proxy-web/0.log"
Jul 10 19:26:15.941643 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.941613    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77c964dcc5-89zt7_b6c7a7f8-98f7-4469-a203-be4c8eb99869/kube-rbac-proxy/0.log"
Jul 10 19:26:15.967784 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.967695    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77c964dcc5-89zt7_b6c7a7f8-98f7-4469-a203-be4c8eb99869/prom-label-proxy/0.log"
Jul 10 19:26:15.992983 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:15.992950    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77c964dcc5-89zt7_b6c7a7f8-98f7-4469-a203-be4c8eb99869/kube-rbac-proxy-rules/0.log"
Jul 10 19:26:16.025405 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:16.025376    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77c964dcc5-89zt7_b6c7a7f8-98f7-4469-a203-be4c8eb99869/kube-rbac-proxy-metrics/0.log"
Jul 10 19:26:18.099085 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.099049    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d4ccf654f-7v6t7_98284170-bdd0-4e98-bd5c-cb3d28fe396c/console/0.log"
Jul 10 19:26:18.131185 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.131160    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-96gct_116f558a-6e70-474a-83d8-fa124957af41/download-server/0.log"
Jul 10 19:26:18.517692 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.517607    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"]
Jul 10 19:26:18.518128 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.518103    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" containerName="copy"
Jul 10 19:26:18.518128 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.518128    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" containerName="copy"
Jul 10 19:26:18.518314 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.518180    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" containerName="gather"
Jul 10 19:26:18.518314 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.518190    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" containerName="gather"
Jul 10 19:26:18.518314 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.518269    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" containerName="copy"
Jul 10 19:26:18.518314 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.518288    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="197beb33-0a14-4306-a989-de5d5c3cf07c" containerName="gather"
Jul 10 19:26:18.522410 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.522376    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.533132 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.533100    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"]
Jul 10 19:26:18.561983 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.561893    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-lib-modules\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.561983 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.561970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-podres\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.562194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.562018    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-sys\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.562194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.562066    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2nmk\" (UniqueName: \"kubernetes.io/projected/e731e3ff-7d54-4b16-af39-cd27d1b87f10-kube-api-access-b2nmk\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.562194 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.562109    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-proc\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.663381 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.663354    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-podres\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.663551 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.663405    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-sys\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.663551 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.663449    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2nmk\" (UniqueName: \"kubernetes.io/projected/e731e3ff-7d54-4b16-af39-cd27d1b87f10-kube-api-access-b2nmk\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.663551 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.663524    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-podres\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.663705 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.663545    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-sys\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.663705 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.663584    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-proc\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.663705 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.663673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-lib-modules\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.663841 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.663674    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-proc\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.663841 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.663774    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e731e3ff-7d54-4b16-af39-cd27d1b87f10-lib-modules\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.672305 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.672280    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2nmk\" (UniqueName: \"kubernetes.io/projected/e731e3ff-7d54-4b16-af39-cd27d1b87f10-kube-api-access-b2nmk\") pod \"perf-node-gather-daemonset-l6nxd\" (UID: \"e731e3ff-7d54-4b16-af39-cd27d1b87f10\") " pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:18.838231 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:18.838196    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:19.004837 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:19.004345    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"]
Jul 10 19:26:19.012694 ip-10-0-136-15 kubenswrapper[2567]: W0710 19:26:19.012667    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode731e3ff_7d54_4b16_af39_cd27d1b87f10.slice/crio-ff62bc4f5d618b4d7603214d3d3f4f2f7f7293e72d65ca89c2baf0d8a7803400 WatchSource:0}: Error finding container ff62bc4f5d618b4d7603214d3d3f4f2f7f7293e72d65ca89c2baf0d8a7803400: Status 404 returned error can't find the container with id ff62bc4f5d618b4d7603214d3d3f4f2f7f7293e72d65ca89c2baf0d8a7803400
Jul 10 19:26:19.448533 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:19.448450    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd" event={"ID":"e731e3ff-7d54-4b16-af39-cd27d1b87f10","Type":"ContainerStarted","Data":"c4e8e233dd8065fd6952bcfd9c01eb500ad0bbdb29c672c75ecfff818a61df71"}
Jul 10 19:26:19.448533 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:19.448501    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd" event={"ID":"e731e3ff-7d54-4b16-af39-cd27d1b87f10","Type":"ContainerStarted","Data":"ff62bc4f5d618b4d7603214d3d3f4f2f7f7293e72d65ca89c2baf0d8a7803400"}
Jul 10 19:26:19.448940 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:19.448594    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:19.470049 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:19.470003    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd" podStartSLOduration=1.469989715 podStartE2EDuration="1.469989715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:26:19.469047682 +0000 UTC m=+1682.720695013" watchObservedRunningTime="2026-07-10 19:26:19.469989715 +0000 UTC m=+1682.721637045"
Jul 10 19:26:19.524846 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:19.524815    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bx94k_7646e080-f86f-411c-93b0-b57ccfb25013/dns/0.log"
Jul 10 19:26:19.547010 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:19.546970    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bx94k_7646e080-f86f-411c-93b0-b57ccfb25013/kube-rbac-proxy/0.log"
Jul 10 19:26:19.679479 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:19.679406    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4xtcp_ebfcae4d-1a86-4c05-b134-59ad0ec54c15/dns-node-resolver/0.log"
Jul 10 19:26:20.260607 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:20.260577    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m9r7c_41ebb4c5-aab3-47f9-8c72-9c7d4ffe1a54/node-ca/0.log"
Jul 10 19:26:21.459419 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:21.459386    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hrppr_ecff79fd-0d2f-4d7d-8335-63b2b884c39a/serve-healthcheck-canary/0.log"
Jul 10 19:26:21.900029 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:21.900004    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tksk4_689d980b-bb4b-4e18-86a1-eaea6f78ba18/kube-rbac-proxy/0.log"
Jul 10 19:26:21.920885 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:21.920859    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tksk4_689d980b-bb4b-4e18-86a1-eaea6f78ba18/exporter/0.log"
Jul 10 19:26:21.943025 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:21.942999    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tksk4_689d980b-bb4b-4e18-86a1-eaea6f78ba18/extractor/0.log"
Jul 10 19:26:24.075024 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:24.074994    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-98r84_6f744953-d7b3-45a0-8268-e4e75c7c2feb/manager/0.log"
Jul 10 19:26:24.166764 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:24.166735    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-lqj7q_617f3522-3d9d-4800-9461-4dc7628bf8c5/manager/0.log"
Jul 10 19:26:24.197033 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:24.197005    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-xbg6x_3f4ef870-3dc2-46e0-960a-ba52569f9ee9/s3-init/0.log"
Jul 10 19:26:24.213007 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:24.212974    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-xbg6x_3f4ef870-3dc2-46e0-960a-ba52569f9ee9/download-hf-model/0.log"
Jul 10 19:26:24.241384 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:24.241357    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-nqnp7_c6832344-e91d-457f-abdc-e363b71c7f7f/s3-tls-init-custom/0.log"
Jul 10 19:26:24.258281 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:24.258247    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-nqnp7_c6832344-e91d-457f-abdc-e363b71c7f7f/download-hf-model/0.log"
Jul 10 19:26:24.289831 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:24.289807    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-qjdxk_fc4c6cad-414d-4215-aabe-44a039cec360/s3-tls-init-serving/0.log"
Jul 10 19:26:24.308419 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:24.308396    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-qjdxk_fc4c6cad-414d-4215-aabe-44a039cec360/download-hf-model/0.log"
Jul 10 19:26:25.464007 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:25.463980    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-znhzn/perf-node-gather-daemonset-l6nxd"
Jul 10 19:26:29.611120 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:29.611085    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5hnlc_cbd3893d-7ab5-4d0c-b7e4-39a0e66ab1c4/kube-multus/0.log"
Jul 10 19:26:29.637882 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:29.637856    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m2j6_354e5926-7524-4d50-a294-6d30003bc759/kube-multus-additional-cni-plugins/0.log"
Jul 10 19:26:29.658413 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:29.658389    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m2j6_354e5926-7524-4d50-a294-6d30003bc759/egress-router-binary-copy/0.log"
Jul 10 19:26:29.680158 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:29.680136    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m2j6_354e5926-7524-4d50-a294-6d30003bc759/cni-plugins/0.log"
Jul 10 19:26:29.700191 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:29.700172    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m2j6_354e5926-7524-4d50-a294-6d30003bc759/bond-cni-plugin/0.log"
Jul 10 19:26:29.722112 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:29.722057    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m2j6_354e5926-7524-4d50-a294-6d30003bc759/routeoverride-cni/0.log"
Jul 10 19:26:29.742561 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:29.742544    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m2j6_354e5926-7524-4d50-a294-6d30003bc759/whereabouts-cni-bincopy/0.log"
Jul 10 19:26:29.762992 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:29.762969    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m2j6_354e5926-7524-4d50-a294-6d30003bc759/whereabouts-cni/0.log"
Jul 10 19:26:30.211659 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:30.211616    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cbt5f_c553a64a-aa13-4a33-90c3-aee6b4f75207/network-metrics-daemon/0.log"
Jul 10 19:26:30.230376 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:30.230352    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cbt5f_c553a64a-aa13-4a33-90c3-aee6b4f75207/kube-rbac-proxy/0.log"
Jul 10 19:26:30.946817 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:30.946791    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdfcs_3bbdd3e9-01c4-4253-8fec-d519904e4e08/ovn-controller/0.log"
Jul 10 19:26:30.963140 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:30.963113    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdfcs_3bbdd3e9-01c4-4253-8fec-d519904e4e08/ovn-acl-logging/0.log"
Jul 10 19:26:30.986635 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:30.986616    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdfcs_3bbdd3e9-01c4-4253-8fec-d519904e4e08/kube-rbac-proxy-node/0.log"
Jul 10 19:26:31.007207 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:31.007192    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdfcs_3bbdd3e9-01c4-4253-8fec-d519904e4e08/kube-rbac-proxy-ovn-metrics/0.log"
Jul 10 19:26:31.028270 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:31.028254    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdfcs_3bbdd3e9-01c4-4253-8fec-d519904e4e08/northd/0.log"
Jul 10 19:26:31.053038 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:31.052999    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdfcs_3bbdd3e9-01c4-4253-8fec-d519904e4e08/nbdb/0.log"
Jul 10 19:26:31.073148 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:31.073126    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdfcs_3bbdd3e9-01c4-4253-8fec-d519904e4e08/sbdb/0.log"
Jul 10 19:26:31.120797 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:31.120774    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdfcs_3bbdd3e9-01c4-4253-8fec-d519904e4e08/ovnkube-controller/0.log"
Jul 10 19:26:32.593013 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:32.592984    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-mgkkh_76040466-2342-4008-becd-8b68f1f99e5e/network-check-target-container/0.log"
Jul 10 19:26:33.421028 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:33.421000    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-72vr9_aab81ea0-5cd4-4e18-b48e-70b3a18292b2/iptables-alerter/0.log"
Jul 10 19:26:34.056775 ip-10-0-136-15 kubenswrapper[2567]: I0710 19:26:34.056749    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-6wg9f_389d4ea0-eee9-400b-a8c7-58ff953732d4/tuned/0.log"