Jul 20 09:55:22.048595 ip-10-0-129-35 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 20 09:55:22.048608 ip-10-0-129-35 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 20 09:55:22.048616 ip-10-0-129-35 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 20 09:55:22.048935 ip-10-0-129-35 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 20 09:55:33.196584 ip-10-0-129-35 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 20 09:55:33.196600 ip-10-0-129-35 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5d829c89fe164133933508dfa0a48ff4 --
Jul 20 09:57:06.819609 ip-10-0-129-35 systemd[1]: Starting Kubernetes Kubelet...
Jul 20 09:57:07.272411 ip-10-0-129-35 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:57:07.272411 ip-10-0-129-35 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 20 09:57:07.272411 ip-10-0-129-35 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:57:07.272411 ip-10-0-129-35 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 20 09:57:07.272411 ip-10-0-129-35 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:57:07.273461 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.273377    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 20 09:57:07.281290 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281262    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:57:07.281290 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281282    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:57:07.281290 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281286    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:57:07.281290 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281290    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:57:07.281290 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281293    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:57:07.281290 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281296    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:57:07.281290 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281299    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:57:07.281290 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281302    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281304    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281308    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281311    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281314    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281316    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281319    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281322    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281325    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281327    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281330    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281332    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281335    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281339    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281343    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281346    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281349    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281352    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281355    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281357    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:57:07.281574 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281360    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281363    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281366    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281369    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281373    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281376    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281378    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281381    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281383    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281386    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281389    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281391    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281394    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281396    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281400    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281403    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281406    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281409    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281412    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:57:07.282067 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281416    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281421    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281424    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281427    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281430    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281432    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281435    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281438    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281440    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281443    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281448    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281452    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281455    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281457    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281460    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281462    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281465    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281468    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281470    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:57:07.282552 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281473    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281475    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281478    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281481    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281483    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281486    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281489    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281492    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281495    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281498    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281501    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281503    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281507    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281509    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281512    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281515    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281517    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281520    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281523    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281526    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:57:07.283021 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281529    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281531    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281533    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281536    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281538    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281541    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281543    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281546    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281548    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281551    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281553    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281556    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281558    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.281561    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282393    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282402    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282405    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282408    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282411    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282414    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:57:07.283520 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282418    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282421    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282424    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282427    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282429    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282432    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282435    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282438    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282441    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282443    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282447    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282461    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282464    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282467    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282469    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282472    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282475    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282478    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282481    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282483    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:57:07.284001 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282486    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282488    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282491    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282494    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282496    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282499    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282502    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282505    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282507    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282509    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282512    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282514    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282520    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282523    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282528    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282531    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282534    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282537    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282540    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282543    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:57:07.284562 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282546    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282549    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282551    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282554    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282564    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282567    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282569    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282572    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282575    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282578    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282580    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282583    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282585    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282588    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282590    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282593    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282595    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282599    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282601    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:57:07.285054 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282604    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282606    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282609    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282611    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282614    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282617    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282620    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282622    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282625    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282628    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282630    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282635    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282637    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282640    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282642    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282645    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282648    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282652    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282661    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:57:07.285546 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282664    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282683    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282687    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282690    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282693    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282696    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282699    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282702    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282705    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282708    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282711    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282714    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282717    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282719    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.282722    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283132    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283137    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283139    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283142    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283145    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:57:07.286024 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283148    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283150    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283153    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283155    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283160    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283163    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283167    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283170    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283173    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283175    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283178    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283186    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283189    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283192    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283194    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283197    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283200    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283202    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283205    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:57:07.286532 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283207    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283210    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283214    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283216    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283219    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283222    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283224    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283226    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283229    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283232    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283234    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283237    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283240    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283258    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283261    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283264    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283267    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283269    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283272    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283275    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:57:07.287000 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283277    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283280    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283283    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283285    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283288    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283291    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283294    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283296    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283300    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283303    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283306    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283308    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283311    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283313    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283318    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283321    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283323    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283326    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283328    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:57:07.287494 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283331    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283334    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283336    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283339    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283344    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283347    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283349    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283352    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283355    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283357    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283360    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283364    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283367    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283370    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283372    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283375    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283377    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283380    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283382    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283385    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:57:07.287964 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283388    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283391    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283393    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283396    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283398    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283401    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283403    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283407    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283410    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283412    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283414    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283417    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283420    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283423    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283425    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.283427    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283504    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283519    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283526    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283531    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283535    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 20 09:57:07.288458 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283539    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283544    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283548    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283552    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283555    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283558    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283562    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283566    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283569    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283572    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283575    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283579    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283582    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283588    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283591    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283594    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283597    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283600    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283605    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283608    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283611    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283614    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283618    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283621    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 20 09:57:07.288969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283624    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283627    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283630    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283635    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283639    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283641    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283644    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283647    2555 flags.go:64] FLAG: --enable-server="true"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283650    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283655    2555 flags.go:64] FLAG: --event-burst="100"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283658    2555 flags.go:64] FLAG: --event-qps="50"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283661    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283664    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283667    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283672    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283675    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283677    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283681    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283683    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283686    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283690    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283693    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283696    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283699    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283701    2555 flags.go:64] FLAG: --feature-gates=""
Jul 20 09:57:07.289582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283705    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283709    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283712    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283715    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283719    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283722    2555 flags.go:64] FLAG: --help="false"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283725    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283728    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283731    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283733    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283738    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283742    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283745    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283748    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283751    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283754    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283757    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283760    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283763    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283766    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283769    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283772    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283775    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283778    2555 flags.go:64] FLAG: --lock-file=""
Jul 20 09:57:07.290307 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283781    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283784    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283787    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283792    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283796    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283799    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283802    2555 flags.go:64] FLAG: --logging-format="text"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283805    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283808    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283811    2555 flags.go:64] FLAG: --manifest-url=""
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283814    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283818    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283821    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283825    2555 flags.go:64] FLAG: --max-pods="110"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283829    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283831    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283834    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283837    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283842    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283845    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283848    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283856    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283860    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283863    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 20 09:57:07.290896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283866    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283870    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283875    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283878    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283881    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283884    2555 flags.go:64] FLAG: --port="10250"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283887    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283890    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02b9ddd1cfb8288a7"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283894    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283897    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283899    2555 flags.go:64] FLAG: --register-node="true"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283902    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283906    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283910    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283913    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283915    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283919    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283922    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283925    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283928    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283931    2555 flags.go:64] FLAG: --runonce="false"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283934    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283937    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283940    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283943    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 20 09:57:07.291539 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283946    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283949    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283953    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283957    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283960    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283963    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283965    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283968    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283971    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283974    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283977    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283983    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283985    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283988    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283993    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283996    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.283998    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.284002    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.284005    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.284009    2555 flags.go:64] FLAG: --v="2"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.284013    2555 flags.go:64] FLAG: --version="false"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.284018    2555 flags.go:64] FLAG: --vmodule=""
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.284021    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.284025    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284149    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:57:07.292159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284153    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284156    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284159    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284162    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284164    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284167    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284169    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284172    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284175    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284179    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284182    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284184    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284187    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284190    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284192    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284195    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284198    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284200    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284203    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284205    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:57:07.292846 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284208    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284211    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284213    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284216    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284219    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284222    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284224    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284227    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284230    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284238    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284241    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284259    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284262    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284264    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284267    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284270    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284272    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284275    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284277    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:57:07.293351 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284280    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284283    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284287    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284290    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284293    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284296    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284298    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284301    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284303    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284307    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284309    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284312    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284315    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284317    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284320    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284323    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284325    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284328    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284330    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284333    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:57:07.293821 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284336    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284338    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284341    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284345    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284349    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284351    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284354    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284356    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284359    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284361    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284365    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284369    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284372    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284375    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284379    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284382    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284386    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284389    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284392    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:57:07.294342 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284395    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284398    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284400    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284403    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284405    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284408    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284411    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284414    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284416    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284419    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284421    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284424    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284427    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284430    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284434    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284437    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284440    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284445    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284448    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:57:07.294809 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.284451    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.285275    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.292813    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.292832    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.292853    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.292859    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292946    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292966    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292970    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292974    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292977    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292980    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292983    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292986    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292990    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:57:07.295321 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292992    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.292997    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293001    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293004    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293006    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293009    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293012    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293015    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293018    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293020    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293023    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293026    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293028    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293031    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293033    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293037    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293040    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293042    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293045    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:57:07.295712 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293047    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293051    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293054    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293056    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293059    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293062    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293064    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293067    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293070    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293072    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293075    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293078    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293080    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293083    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293085    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293088    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293091    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293093    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293096    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293099    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:57:07.296159 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293101    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293104    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293106    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293109    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293112    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293115    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293117    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293120    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293122    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293126    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293129    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293132    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293134    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293137    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293140    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293142    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293145    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293147    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293149    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293152    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:57:07.296667 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293155    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293157    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293160    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293163    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293166    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293169    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293171    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293174    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293176    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293179    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293181    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293184    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293186    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293189    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293191    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293194    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293199    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293202    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293205    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:57:07.297150 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293208    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293211    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293214    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293217    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293220    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293223    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293225    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293228    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293231    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293234    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293238    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293257    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.293264    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293401    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293406    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:57:07.297644 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293409    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293413    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293416    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293418    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293421    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293425    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293428    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293431    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293433    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293436    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293439    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293441    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293444    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293446    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293449    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293451    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293454    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293457    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293459    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293462    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:57:07.298040 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293464    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293467    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293470    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293473    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293476    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293479    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293482    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293484    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293487    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293490    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293492    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293495    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293497    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293500    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293502    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293505    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293507    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293510    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293513    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293515    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:57:07.298545 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293518    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293520    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293523    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293526    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293534    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293537    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293540    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293542    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293544    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293547    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293550    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293552    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293554    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293557    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293565    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293568    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293571    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293574    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293576    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:57:07.299033 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293579    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293581    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293584    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293586    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293589    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293591    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293594    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293597    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293599    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293601    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293604    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293606    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293609    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293611    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293614    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293617    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293619    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293621    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293624    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293627    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:57:07.299679 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293630    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293633    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293636    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293638    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293642    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293645    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293648    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293651    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293661    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293664    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293666    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293669    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293672    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293674    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293677    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293679    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293682    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:57:07.300158 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:07.293686    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:57:07.300585 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.293691    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:57:07.300585 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.293818    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 20 09:57:07.300585 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.296342    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 20 09:57:07.300585 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.298090    2555 server.go:1019] "Starting client certificate rotation"
Jul 20 09:57:07.300585 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.298197    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:57:07.300585 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.298238    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:57:07.301079 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.301063    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:57:07.305473 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.305447    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:57:07.317044 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.317023    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 20 09:57:07.321430 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.321409    2555 log.go:25] "Validated CRI v1 image API"
Jul 20 09:57:07.322213 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.322197    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 20 09:57:07.324905 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.324879    2555 fs.go:135] Filesystem UUIDs: map[63bea3f3-70b6-4ef9-a4c9-d4881b5333c7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a6610d64-6cc4-4c6f-80b7-d6ca4c00f74b:/dev/nvme0n1p3]
Jul 20 09:57:07.324970 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.324905    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 20 09:57:07.332312 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.332169    2555 manager.go:216] Machine: {Timestamp:2026-07-20 09:57:07.330698261 +0000 UTC m=+0.381627607 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100067 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d06ea4f5b55c831e7d49eb510aedf SystemUUID:ec2d06ea-4f5b-55c8-31e7-d49eb510aedf BootID:5d829c89-fe16-4133-9335-08dfa0a48ff4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:69:19:c8:92:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:69:19:c8:92:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:58:c2:74:83:84 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 20 09:57:07.332312 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.332301    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 20 09:57:07.332474 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.332439    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 20 09:57:07.332573 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.332555    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:57:07.335888 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.335859    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 20 09:57:07.336057 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.335890    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-35.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 20 09:57:07.336754 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.336743    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 20 09:57:07.336793 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.336757    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 20 09:57:07.336793 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.336764    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:57:07.336793 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.336778    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:57:07.338702 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.338689    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 20 09:57:07.343490 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.343474    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:57:07.343672 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.343662    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 20 09:57:07.343716 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.343695    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 20 09:57:07.343716 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.343714    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 20 09:57:07.343787 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.343726    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 20 09:57:07.343787 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.343736    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 20 09:57:07.343787 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.343748    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 20 09:57:07.344142 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.344130    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:57:07.344188 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.344148    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:57:07.346621 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.346600    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 20 09:57:07.349358 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.349340    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 20 09:57:07.349426 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.349400    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 20 09:57:07.350866 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350853    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 20 09:57:07.350905 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350870    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 20 09:57:07.350905 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350877    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 20 09:57:07.350905 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350884    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 20 09:57:07.350905 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350898    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 20 09:57:07.350905 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350906    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 20 09:57:07.351036 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350916    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 20 09:57:07.351036 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350921    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 20 09:57:07.351036 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350928    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 20 09:57:07.351036 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350935    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 20 09:57:07.351036 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350949    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 20 09:57:07.351036 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.350955    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 20 09:57:07.351036 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.351002    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 20 09:57:07.351036 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.351008    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 20 09:57:07.354796 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.354780    2555 server.go:1268] "Started kubelet"
Jul 20 09:57:07.354950 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.354878    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 20 09:57:07.355013 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.354929    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 20 09:57:07.355070 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.355031    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 20 09:57:07.355727 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.355691    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-35.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 20 09:57:07.355814 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.355727    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 20 09:57:07.355956 ip-10-0-129-35 systemd[1]: Started Kubernetes Kubelet.
Jul 20 09:57:07.356138 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.356103    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 20 09:57:07.357288 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.357270    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 20 09:57:07.358466 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.358450    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 20 09:57:07.364175 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.359497    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-35.ec2.internal.18c3f6f928721f82  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-35.ec2.internal,UID:ip-10-0-129-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-35.ec2.internal,},FirstTimestamp:2026-07-20 09:57:07.354705794 +0000 UTC m=+0.405635140,LastTimestamp:2026-07-20 09:57:07.354705794 +0000 UTC m=+0.405635140,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-35.ec2.internal,}"
Jul 20 09:57:07.364297 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.364159    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rr675"
Jul 20 09:57:07.364697 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.364665    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 20 09:57:07.368005 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.367981    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rr675"
Jul 20 09:57:07.369380 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.369359    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 20 09:57:07.369520 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.369504    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 20 09:57:07.369632 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.369550    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-35.ec2.internal\" not found"
Jul 20 09:57:07.370199 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370182    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 20 09:57:07.370199 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370200    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 20 09:57:07.370358 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370262    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 20 09:57:07.370476 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370460    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 20 09:57:07.370476 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370474    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 20 09:57:07.370606 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370590    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 20 09:57:07.370606 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370606    2555 factory.go:55] Registering systemd factory
Jul 20 09:57:07.370701 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370613    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 20 09:57:07.370842 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370829    2555 factory.go:153] Registering CRI-O factory
Jul 20 09:57:07.370896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370847    2555 factory.go:223] Registration of the crio container factory successfully
Jul 20 09:57:07.370896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370871    2555 factory.go:103] Registering Raw factory
Jul 20 09:57:07.370896 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.370884    2555 manager.go:1196] Started watching for new ooms in manager
Jul 20 09:57:07.371358 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.371347    2555 manager.go:349] Starting recovery of all containers
Jul 20 09:57:07.380745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.380715    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:07.383421 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.383404    2555 manager.go:354] Recovery completed
Jul 20 09:57:07.386049 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.386030    2555 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-35.ec2.internal\" not found" node="ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.387786 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.387774    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:07.389660 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.389646    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:07.389701 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.389679    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:07.389701 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.389694    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:07.390230 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.390209    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 20 09:57:07.390230 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.390222    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 20 09:57:07.390334 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.390237    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:07.390377 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.390255    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:57:07.390959 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.390940    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:07.391030 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.390971    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:07.391030 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.390989    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:07.392492 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.392475    2555 policy_none.go:49] "None policy: Start"
Jul 20 09:57:07.392492 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.392493    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 20 09:57:07.392627 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.392503    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 20 09:57:07.394530 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.394516    2555 policy_none.go:47] "Start"
Jul 20 09:57:07.430922 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.430894    2555 manager.go:337] "Starting Device Plugin manager"
Jul 20 09:57:07.431110 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.430933    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 20 09:57:07.431110 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.430947    2555 server.go:87] "Starting device plugin registration server"
Jul 20 09:57:07.431306 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.431237    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 20 09:57:07.431361 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.431295    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 20 09:57:07.431441 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.431422    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 20 09:57:07.431534 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.431516    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 20 09:57:07.431534 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.431532    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 20 09:57:07.432408 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.432257    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 20 09:57:07.432408 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.432312    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-35.ec2.internal\" not found"
Jul 20 09:57:07.483987 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.483941    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 20 09:57:07.485208 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.485194    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 20 09:57:07.485300 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.485237    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 20 09:57:07.485300 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.485280    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 20 09:57:07.485400 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.485331    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 20 09:57:07.487960 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.487941    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:07.532356 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.532285    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:07.533367 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.533350    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:07.533460 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.533379    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:07.533460 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.533393    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:07.533460 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.533425    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.552662 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.552632    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.552804 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.552671    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-35.ec2.internal\": node \"ip-10-0-129-35.ec2.internal\" not found"
Jul 20 09:57:07.585150 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.585120    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-35.ec2.internal\" not found"
Jul 20 09:57:07.586681 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.586083    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal"]
Jul 20 09:57:07.586769 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.586724    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:07.587646 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.587631    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:07.587736 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.587661    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:07.587736 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.587671    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:07.590002 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.589989    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:07.590148 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.590134    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.590185 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.590164    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:07.590743 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.590718    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:07.590830 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.590725    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:07.590830 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.590769    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:07.590830 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.590780    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:07.590830 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.590750    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:07.590830 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.590829    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:07.593084 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.593067    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.593191 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.593092    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:07.593794 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.593781    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:07.593877 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.593809    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:07.593877 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.593820    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:07.615935 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.615904    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-35.ec2.internal\" not found" node="ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.619611 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.619588    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-35.ec2.internal\" not found" node="ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.685395 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.685187    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-35.ec2.internal\" not found"
Jul 20 09:57:07.771785 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.771755    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba2adbd0e228b3393b7ea7327f58790b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-35.ec2.internal\" (UID: \"ba2adbd0e228b3393b7ea7327f58790b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.771892 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.771787    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf485a96b13d861618003dd2b5d77147-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal\" (UID: \"bf485a96b13d861618003dd2b5d77147\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.771892 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.771808    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf485a96b13d861618003dd2b5d77147-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal\" (UID: \"bf485a96b13d861618003dd2b5d77147\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.785819 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.785740    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-35.ec2.internal\" not found"
Jul 20 09:57:07.872209 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.872179    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf485a96b13d861618003dd2b5d77147-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal\" (UID: \"bf485a96b13d861618003dd2b5d77147\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.872285 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.872224    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf485a96b13d861618003dd2b5d77147-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal\" (UID: \"bf485a96b13d861618003dd2b5d77147\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.872285 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.872274    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba2adbd0e228b3393b7ea7327f58790b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-35.ec2.internal\" (UID: \"ba2adbd0e228b3393b7ea7327f58790b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.872359 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.872323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba2adbd0e228b3393b7ea7327f58790b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-35.ec2.internal\" (UID: \"ba2adbd0e228b3393b7ea7327f58790b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.872395 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.872349    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf485a96b13d861618003dd2b5d77147-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal\" (UID: \"bf485a96b13d861618003dd2b5d77147\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.872395 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.872323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf485a96b13d861618003dd2b5d77147-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal\" (UID: \"bf485a96b13d861618003dd2b5d77147\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.886391 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.886332    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-35.ec2.internal\" not found"
Jul 20 09:57:07.921159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.921133    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.922324 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:07.922308    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:07.987467 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:07.987407    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-35.ec2.internal\" not found"
Jul 20 09:57:08.088038 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.087936    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-35.ec2.internal\" not found"
Jul 20 09:57:08.188927 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.188891    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-35.ec2.internal\" not found"
Jul 20 09:57:08.196401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.196377    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:08.275732 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.275696    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:08.290327 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.290301    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:57:08.291910 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.291889    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:08.297163 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.297139    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 20 09:57:08.297308 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.297275    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 09:57:08.297432 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.297311    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 09:57:08.297432 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.297315    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 09:57:08.297432 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.297363    2555 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a28580b1d39df4317baae046d3533e69-8ff350f4d834eafb.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.129.35:54144->54.165.173.15:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal"
Jul 20 09:57:08.347415 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.347320    2555 apiserver.go:52] "Watching apiserver"
Jul 20 09:57:08.367749 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.367713    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 20 09:57:08.368774 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.368749    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-q7zvj","openshift-ovn-kubernetes/ovnkube-node-8dsbz","kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq","openshift-cluster-node-tuning-operator/tuned-xzt4t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal","openshift-multus/multus-additional-cni-plugins-68fww","openshift-multus/multus-cnzgt","openshift-multus/network-metrics-daemon-p24w6","openshift-network-operator/iptables-alerter-kpskk","kube-system/global-pull-secret-syncer-5r7zg","kube-system/konnectivity-agent-c2gwd","openshift-dns/node-resolver-tfh95","openshift-image-registry/node-ca-cm5ps"]
Jul 20 09:57:08.370082 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.370060    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 20 09:57:08.371502 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.371484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.373763 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.373743    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.373865 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.373835    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 09:52:07 +0000 UTC" deadline="2027-12-17 07:03:37.296729998 +0000 UTC"
Jul 20 09:57:08.373865 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.373859    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12357h6m28.922875056s"
Jul 20 09:57:08.376159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.376120    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.378418 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.378397    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-os-release\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.378504 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.378430    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bfcd6077-e7cf-4e17-829b-514131357a1a-cni-binary-copy\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.378504 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.378460    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bfcd6077-e7cf-4e17-829b-514131357a1a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.378632 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.378503    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.378632 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.378499    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-system-cni-dir\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.378742 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.378622    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-cnibin\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.378742 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.378679    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.378742 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.378698    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bfcd6077-e7cf-4e17-829b-514131357a1a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.378869 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.378775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5l6\" (UniqueName: \"kubernetes.io/projected/bfcd6077-e7cf-4e17-829b-514131357a1a-kube-api-access-jd5l6\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.379025 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.379006    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 20 09:57:08.379188 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.379166    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 20 09:57:08.379261 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.379206    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 20 09:57:08.379261 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.379240    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 20 09:57:08.379386 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.379297    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 20 09:57:08.379386 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.379319    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 20 09:57:08.379386 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.379177    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 20 09:57:08.379386 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.379325    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 20 09:57:08.379386 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.379241    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 20 09:57:08.379549 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.379472    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 20 09:57:08.380548 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.380531    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vhplr\""
Jul 20 09:57:08.380603 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.380537    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rdh9m\""
Jul 20 09:57:08.380603 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.380598    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jmtnz\""
Jul 20 09:57:08.380940 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.380927    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.383347 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.383326    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.383745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.383726    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k92jd\""
Jul 20 09:57:08.383851 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.383834    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 20 09:57:08.383903 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.383842    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 20 09:57:08.383961 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.383842    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 20 09:57:08.384518 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.384503    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:08.384838 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.384820    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rplxq\""
Jul 20 09:57:08.385264 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.385233    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:08.387948 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.387929    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:08.390415 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.390397    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:08.390499 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.390483    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:08.391258 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.391228    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 20 09:57:08.391674 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.391659    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xhp5n\""
Jul 20 09:57:08.392607 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.392586    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:57:08.393525 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.393306    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.395619 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.395602    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 20 09:57:08.395956 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.395941    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.397036 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.397018    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5xbxt\""
Jul 20 09:57:08.397158 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.397142    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 20 09:57:08.397224 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.397182    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 20 09:57:08.397224 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.397196    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 20 09:57:08.397359 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.397311    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p4zxh\""
Jul 20 09:57:08.397480 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.397464    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 20 09:57:08.397598 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.397577    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 20 09:57:08.398779 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.398760    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 20 09:57:08.398779 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.398771    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:08.398928 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.398851    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:08.399418 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.399402    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:08.399668 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.399652    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:08.399752 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.399688    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x72sh\""
Jul 20 09:57:08.400420 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.400405    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 20 09:57:08.401221 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.401204    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:08.401299 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.401264    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:08.404222 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.404208    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 20 09:57:08.409225 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.409202    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r6r57"
Jul 20 09:57:08.419934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.419894    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r6r57"
Jul 20 09:57:08.428757 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.428725    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:08.471280 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.471258    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 20 09:57:08.479026 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.478999    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-system-cni-dir\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.479186 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5l6\" (UniqueName: \"kubernetes.io/projected/bfcd6077-e7cf-4e17-829b-514131357a1a-kube-api-access-jd5l6\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.479186 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479061    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmvgf\" (UniqueName: \"kubernetes.io/projected/cba89306-92d2-4c9f-a577-d992c350a925-kube-api-access-lmvgf\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.479186 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479086    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2851d0a8-1480-444d-bc7e-ab21239553df-ovn-node-metrics-cert\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.479186 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479104    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-system-cni-dir\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.479186 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479113    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3796601d-5123-4213-8c2d-453d6212d472-kubelet-config\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:08.479186 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:08.479186 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479167    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-sys-fs\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.479186 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479190    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-socket-dir-parent\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-var-lib-kubelet\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-cnibin\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479310    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-modprobe-d\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479336    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8bee537-c2d5-4977-a3fd-b8d6cc021900-konnectivity-ca\") pod \"konnectivity-agent-c2gwd\" (UID: \"d8bee537-c2d5-4977-a3fd-b8d6cc021900\") " pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479345    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-cnibin\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479363    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-slash\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479402    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-device-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479430    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-var-lib-openvswitch\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479454    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2851d0a8-1480-444d-bc7e-ab21239553df-ovnkube-script-lib\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479480    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45z2\" (UniqueName: \"kubernetes.io/projected/a0274aa1-d4b8-463c-8f61-d888fe04e631-kube-api-access-c45z2\") pod \"node-resolver-tfh95\" (UID: \"a0274aa1-d4b8-463c-8f61-d888fe04e631\") " pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.479544 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479511    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-var-lib-cni-bin\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479555    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-run-multus-certs\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479598    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3796601d-5123-4213-8c2d-453d6212d472-dbus\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479627    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479653    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-var-lib-kubelet\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479681    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-log-socket\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479706    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58dc4f4e-0a78-4b1d-af07-30685a4498d0-serviceca\") pod \"node-ca-cm5ps\" (UID: \"58dc4f4e-0a78-4b1d-af07-30685a4498d0\") " pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479760    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cba89306-92d2-4c9f-a577-d992c350a925-etc-tuned\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479785    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-systemd-units\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-node-log\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479832    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-conf-dir\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479857    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx9wb\" (UniqueName: \"kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb\") pod \"network-check-target-q7zvj\" (UID: \"1d7b7476-9151-41f4-a037-54a40c87cdc4\") " pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:08.479943 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479917    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvl5f\" (UniqueName: \"kubernetes.io/projected/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-kube-api-access-gvl5f\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8w49\" (UniqueName: \"kubernetes.io/projected/58dc4f4e-0a78-4b1d-af07-30685a4498d0-kube-api-access-d8w49\") pod \"node-ca-cm5ps\" (UID: \"58dc4f4e-0a78-4b1d-af07-30685a4498d0\") " pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.479973    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-sysctl-d\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8bee537-c2d5-4977-a3fd-b8d6cc021900-agent-certs\") pod \"konnectivity-agent-c2gwd\" (UID: \"d8bee537-c2d5-4977-a3fd-b8d6cc021900\") " pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480067    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-kubelet\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480089    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2851d0a8-1480-444d-bc7e-ab21239553df-env-overrides\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480106    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0274aa1-d4b8-463c-8f61-d888fe04e631-hosts-file\") pod \"node-resolver-tfh95\" (UID: \"a0274aa1-d4b8-463c-8f61-d888fe04e631\") " pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480123    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-var-lib-cni-multus\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480141    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-etc-kubernetes\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480164    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480183    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kfh5\" (UniqueName: \"kubernetes.io/projected/2851d0a8-1480-444d-bc7e-ab21239553df-kube-api-access-8kfh5\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480225    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480280    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-lib-modules\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480308    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-cni-netd\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480346    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-os-release\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.480523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-run-k8s-cni-cncf-io\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.481193 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480450    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-socket-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.481193 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480486    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bfcd6077-e7cf-4e17-829b-514131357a1a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.481193 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-run-openvswitch\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.481193 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480555    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-daemon-config\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.481193 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480581    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdwz7\" (UniqueName: \"kubernetes.io/projected/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-kube-api-access-cdwz7\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.481193 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480624    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-host\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.481193 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.480428    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.481193 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481027    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bfcd6077-e7cf-4e17-829b-514131357a1a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.481488 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481226    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cba89306-92d2-4c9f-a577-d992c350a925-tmp\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.481488 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481278    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-run-systemd\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.481488 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2851d0a8-1480-444d-bc7e-ab21239553df-ovnkube-config\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.481488 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-cni-binary-copy\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.481488 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481381    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b75520ae-fef6-42fe-a019-59a2f6b04a24-iptables-alerter-script\") pod \"iptables-alerter-kpskk\" (UID: \"b75520ae-fef6-42fe-a019-59a2f6b04a24\") " pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.481488 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481407    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-registration-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.481488 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481429    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-run-netns\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.481488 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481452    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-run-ovn\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.481488 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481502    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0274aa1-d4b8-463c-8f61-d888fe04e631-tmp-dir\") pod \"node-resolver-tfh95\" (UID: \"a0274aa1-d4b8-463c-8f61-d888fe04e631\") " pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481534    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-system-cni-dir\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481564    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-run-netns\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481608    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-kubernetes\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481631    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-run\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481649    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-etc-openvswitch\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481667    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-cni-bin\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481701    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-hostroot\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481737    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7x4\" (UniqueName: \"kubernetes.io/projected/b75520ae-fef6-42fe-a019-59a2f6b04a24-kube-api-access-xl7x4\") pod \"iptables-alerter-kpskk\" (UID: \"b75520ae-fef6-42fe-a019-59a2f6b04a24\") " pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-os-release\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481812    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bfcd6077-e7cf-4e17-829b-514131357a1a-cni-binary-copy\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481840    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-sysctl-conf\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481865    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-cnibin\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481875    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bfcd6077-e7cf-4e17-829b-514131357a1a-os-release\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.481904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481894    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bfcd6077-e7cf-4e17-829b-514131357a1a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.482468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481919    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-sysconfig\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.482468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481959    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-systemd\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.482468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.481985    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b75520ae-fef6-42fe-a019-59a2f6b04a24-host-slash\") pod \"iptables-alerter-kpskk\" (UID: \"b75520ae-fef6-42fe-a019-59a2f6b04a24\") " pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.482468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.482011    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58dc4f4e-0a78-4b1d-af07-30685a4498d0-host\") pod \"node-ca-cm5ps\" (UID: \"58dc4f4e-0a78-4b1d-af07-30685a4498d0\") " pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.482468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.482034    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-sys\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.482468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.482060    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-cni-dir\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.482468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.482095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-etc-selinux\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.482468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.482121    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzr4x\" (UniqueName: \"kubernetes.io/projected/e1016a67-9fd8-42c1-a6ac-b7220db66003-kube-api-access-vzr4x\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.482468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.482420    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bfcd6077-e7cf-4e17-829b-514131357a1a-cni-binary-copy\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.482743 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.482489    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bfcd6077-e7cf-4e17-829b-514131357a1a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.493998 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.493967    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 20 09:57:08.497794 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.497765    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5l6\" (UniqueName: \"kubernetes.io/projected/bfcd6077-e7cf-4e17-829b-514131357a1a-kube-api-access-jd5l6\") pod \"multus-additional-cni-plugins-68fww\" (UID: \"bfcd6077-e7cf-4e17-829b-514131357a1a\") " pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.582787 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582755    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-slash\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.582787 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582793    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-device-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582813    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-var-lib-openvswitch\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582839    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2851d0a8-1480-444d-bc7e-ab21239553df-ovnkube-script-lib\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582861    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45z2\" (UniqueName: \"kubernetes.io/projected/a0274aa1-d4b8-463c-8f61-d888fe04e631-kube-api-access-c45z2\") pod \"node-resolver-tfh95\" (UID: \"a0274aa1-d4b8-463c-8f61-d888fe04e631\") " pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-var-lib-cni-bin\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582896    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-slash\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582914    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-run-multus-certs\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582916    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-var-lib-openvswitch\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582939    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3796601d-5123-4213-8c2d-453d6212d472-dbus\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-device-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582969    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582975    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-var-lib-cni-bin\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582992    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-var-lib-kubelet\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.582976    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-run-multus-certs\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.583022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583015    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-log-socket\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583039    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58dc4f4e-0a78-4b1d-af07-30685a4498d0-serviceca\") pod \"node-ca-cm5ps\" (UID: \"58dc4f4e-0a78-4b1d-af07-30685a4498d0\") " pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583062    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cba89306-92d2-4c9f-a577-d992c350a925-etc-tuned\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583088    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-systemd-units\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.583098    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-node-log\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-conf-dir\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583159    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9wb\" (UniqueName: \"kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb\") pod \"network-check-target-q7zvj\" (UID: \"1d7b7476-9151-41f4-a037-54a40c87cdc4\") " pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3796601d-5123-4213-8c2d-453d6212d472-dbus\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.583191    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret podName:3796601d-5123-4213-8c2d-453d6212d472 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:09.083164176 +0000 UTC m=+2.134093536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret") pod "global-pull-secret-syncer-5r7zg" (UID: "3796601d-5123-4213-8c2d-453d6212d472") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583216    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-var-lib-kubelet\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583219    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-systemd-units\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583243    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-log-socket\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583257    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-node-log\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583320    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-conf-dir\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583324    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvl5f\" (UniqueName: \"kubernetes.io/projected/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-kube-api-access-gvl5f\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583356    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8w49\" (UniqueName: \"kubernetes.io/projected/58dc4f4e-0a78-4b1d-af07-30685a4498d0-kube-api-access-d8w49\") pod \"node-ca-cm5ps\" (UID: \"58dc4f4e-0a78-4b1d-af07-30685a4498d0\") " pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.583745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583381    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-sysctl-d\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583405    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8bee537-c2d5-4977-a3fd-b8d6cc021900-agent-certs\") pod \"konnectivity-agent-c2gwd\" (UID: \"d8bee537-c2d5-4977-a3fd-b8d6cc021900\") " pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583430    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-kubelet\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583455    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2851d0a8-1480-444d-bc7e-ab21239553df-env-overrides\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583480    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0274aa1-d4b8-463c-8f61-d888fe04e631-hosts-file\") pod \"node-resolver-tfh95\" (UID: \"a0274aa1-d4b8-463c-8f61-d888fe04e631\") " pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583505    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-var-lib-cni-multus\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583532    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-etc-kubernetes\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583557    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kfh5\" (UniqueName: \"kubernetes.io/projected/2851d0a8-1480-444d-bc7e-ab21239553df-kube-api-access-8kfh5\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-lib-modules\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583637    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-cni-netd\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583632    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-sysctl-d\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583659    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2851d0a8-1480-444d-bc7e-ab21239553df-ovnkube-script-lib\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583703    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-var-lib-cni-multus\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583726    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58dc4f4e-0a78-4b1d-af07-30685a4498d0-serviceca\") pod \"node-ca-cm5ps\" (UID: \"58dc4f4e-0a78-4b1d-af07-30685a4498d0\") " pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583764    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0274aa1-d4b8-463c-8f61-d888fe04e631-hosts-file\") pod \"node-resolver-tfh95\" (UID: \"a0274aa1-d4b8-463c-8f61-d888fe04e631\") " pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583769    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-os-release\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.584663 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583796    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-run-k8s-cni-cncf-io\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-kubelet\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583809    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583822    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-socket-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583815    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583853    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-cni-netd\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583869    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-run-k8s-cni-cncf-io\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583897    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-etc-kubernetes\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583908    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-run-openvswitch\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583912    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-lib-modules\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583939    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-daemon-config\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583964    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-os-release\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583969    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdwz7\" (UniqueName: \"kubernetes.io/projected/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-kube-api-access-cdwz7\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583964    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-run-openvswitch\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583974    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-socket-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.583999    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-host\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584027    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cba89306-92d2-4c9f-a577-d992c350a925-tmp\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-host\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.585617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584138    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-run-systemd\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584151    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2851d0a8-1480-444d-bc7e-ab21239553df-env-overrides\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2851d0a8-1480-444d-bc7e-ab21239553df-ovnkube-config\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584216    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-cni-binary-copy\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584218    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-run-systemd\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584258    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b75520ae-fef6-42fe-a019-59a2f6b04a24-iptables-alerter-script\") pod \"iptables-alerter-kpskk\" (UID: \"b75520ae-fef6-42fe-a019-59a2f6b04a24\") " pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-registration-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584314    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-run-netns\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-run-ovn\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584384    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584412    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0274aa1-d4b8-463c-8f61-d888fe04e631-tmp-dir\") pod \"node-resolver-tfh95\" (UID: \"a0274aa1-d4b8-463c-8f61-d888fe04e631\") " pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584440    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-system-cni-dir\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584487    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-run-netns\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584513    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-kubernetes\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584534    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584541    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-run\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584587    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-etc-openvswitch\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584596    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-run\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.586616 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584618    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-cni-bin\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584647    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-hostroot\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584652    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-registration-dir\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584682    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7x4\" (UniqueName: \"kubernetes.io/projected/b75520ae-fef6-42fe-a019-59a2f6b04a24-kube-api-access-xl7x4\") pod \"iptables-alerter-kpskk\" (UID: \"b75520ae-fef6-42fe-a019-59a2f6b04a24\") " pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584698    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-run-netns\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584713    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-sysctl-conf\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584739    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-cnibin\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584757    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-run-ovn\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-sysconfig\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-etc-openvswitch\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584808    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2851d0a8-1480-444d-bc7e-ab21239553df-host-cni-bin\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584810    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-systemd\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584850    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b75520ae-fef6-42fe-a019-59a2f6b04a24-host-slash\") pod \"iptables-alerter-kpskk\" (UID: \"b75520ae-fef6-42fe-a019-59a2f6b04a24\") " pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-run-netns\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584869    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-system-cni-dir\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58dc4f4e-0a78-4b1d-af07-30685a4498d0-host\") pod \"node-ca-cm5ps\" (UID: \"58dc4f4e-0a78-4b1d-af07-30685a4498d0\") " pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584906    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-sys\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584903    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-hostroot\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.587686 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584932    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-kubernetes\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-cni-dir\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584977    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-etc-selinux\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.584993    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-cni-dir\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585004    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzr4x\" (UniqueName: \"kubernetes.io/projected/e1016a67-9fd8-42c1-a6ac-b7220db66003-kube-api-access-vzr4x\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585039    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmvgf\" (UniqueName: \"kubernetes.io/projected/cba89306-92d2-4c9f-a577-d992c350a925-kube-api-access-lmvgf\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585065    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2851d0a8-1480-444d-bc7e-ab21239553df-ovn-node-metrics-cert\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585091    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3796601d-5123-4213-8c2d-453d6212d472-kubelet-config\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585116    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585144    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-sys-fs\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585188    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-socket-dir-parent\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585223    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-var-lib-kubelet\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585249    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-modprobe-d\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585287    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8bee537-c2d5-4977-a3fd-b8d6cc021900-konnectivity-ca\") pod \"konnectivity-agent-c2gwd\" (UID: \"d8bee537-c2d5-4977-a3fd-b8d6cc021900\") " pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585365    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-sysctl-conf\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585438    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-cnibin\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.585461    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf485a96b13d861618003dd2b5d77147.slice/crio-35102e7b071786b3292ebc8f3fa7193d2cef5cb94fe6e1b958f3c05dc1fde4ed WatchSource:0}: Error finding container 35102e7b071786b3292ebc8f3fa7193d2cef5cb94fe6e1b958f3c05dc1fde4ed: Status 404 returned error can't find the container with id 35102e7b071786b3292ebc8f3fa7193d2cef5cb94fe6e1b958f3c05dc1fde4ed
Jul 20 09:57:08.588401 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585498    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-sysconfig\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585505    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2851d0a8-1480-444d-bc7e-ab21239553df-ovnkube-config\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585545    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-systemd\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585592    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b75520ae-fef6-42fe-a019-59a2f6b04a24-host-slash\") pod \"iptables-alerter-kpskk\" (UID: \"b75520ae-fef6-42fe-a019-59a2f6b04a24\") " pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585635    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58dc4f4e-0a78-4b1d-af07-30685a4498d0-host\") pod \"node-ca-cm5ps\" (UID: \"58dc4f4e-0a78-4b1d-af07-30685a4498d0\") " pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585636    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3796601d-5123-4213-8c2d-453d6212d472-kubelet-config\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585682    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-sys\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585818    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-etc-selinux\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585888    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-socket-dir-parent\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8bee537-c2d5-4977-a3fd-b8d6cc021900-konnectivity-ca\") pod \"konnectivity-agent-c2gwd\" (UID: \"d8bee537-c2d5-4977-a3fd-b8d6cc021900\") " pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.585970    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585989    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-host-var-lib-kubelet\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.585994    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1016a67-9fd8-42c1-a6ac-b7220db66003-sys-fs\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.586034    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs podName:ed36d59b-3fc9-43ea-9506-f28d6955b7a3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:09.086015178 +0000 UTC m=+2.136944513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs") pod "network-metrics-daemon-p24w6" (UID: "ed36d59b-3fc9-43ea-9506-f28d6955b7a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.586078    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cba89306-92d2-4c9f-a577-d992c350a925-etc-modprobe-d\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.586586    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-cni-binary-copy\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.586732    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b75520ae-fef6-42fe-a019-59a2f6b04a24-iptables-alerter-script\") pod \"iptables-alerter-kpskk\" (UID: \"b75520ae-fef6-42fe-a019-59a2f6b04a24\") " pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.588934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.586825    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-multus-daemon-config\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.589535 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.586835    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0274aa1-d4b8-463c-8f61-d888fe04e631-tmp-dir\") pod \"node-resolver-tfh95\" (UID: \"a0274aa1-d4b8-463c-8f61-d888fe04e631\") " pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.589535 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.586987    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cba89306-92d2-4c9f-a577-d992c350a925-tmp\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.589535 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.587025    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cba89306-92d2-4c9f-a577-d992c350a925-etc-tuned\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.589535 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.587156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8bee537-c2d5-4977-a3fd-b8d6cc021900-agent-certs\") pod \"konnectivity-agent-c2gwd\" (UID: \"d8bee537-c2d5-4977-a3fd-b8d6cc021900\") " pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:08.589535 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.588234    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2851d0a8-1480-444d-bc7e-ab21239553df-ovn-node-metrics-cert\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.592187 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.592171    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 09:57:08.598275 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.598210    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45z2\" (UniqueName: \"kubernetes.io/projected/a0274aa1-d4b8-463c-8f61-d888fe04e631-kube-api-access-c45z2\") pod \"node-resolver-tfh95\" (UID: \"a0274aa1-d4b8-463c-8f61-d888fe04e631\") " pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.601670 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.601639    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:08.601670 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.601665    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:08.601855 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.601682    2555 projected.go:196] Error preparing data for projected volume kube-api-access-mx9wb for pod openshift-network-diagnostics/network-check-target-q7zvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:08.601855 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:08.601764    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb podName:1d7b7476-9151-41f4-a037-54a40c87cdc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:09.101740278 +0000 UTC m=+2.152669634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mx9wb" (UniqueName: "kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb") pod "network-check-target-q7zvj" (UID: "1d7b7476-9151-41f4-a037-54a40c87cdc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:08.602357 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.602300    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7x4\" (UniqueName: \"kubernetes.io/projected/b75520ae-fef6-42fe-a019-59a2f6b04a24-kube-api-access-xl7x4\") pod \"iptables-alerter-kpskk\" (UID: \"b75520ae-fef6-42fe-a019-59a2f6b04a24\") " pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.602731 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.602701    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvl5f\" (UniqueName: \"kubernetes.io/projected/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-kube-api-access-gvl5f\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:08.602997 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.602970    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdwz7\" (UniqueName: \"kubernetes.io/projected/2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac-kube-api-access-cdwz7\") pod \"multus-cnzgt\" (UID: \"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac\") " pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.603682 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.603660    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmvgf\" (UniqueName: \"kubernetes.io/projected/cba89306-92d2-4c9f-a577-d992c350a925-kube-api-access-lmvgf\") pod \"tuned-xzt4t\" (UID: \"cba89306-92d2-4c9f-a577-d992c350a925\") " pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.604498 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.604472    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8w49\" (UniqueName: \"kubernetes.io/projected/58dc4f4e-0a78-4b1d-af07-30685a4498d0-kube-api-access-d8w49\") pod \"node-ca-cm5ps\" (UID: \"58dc4f4e-0a78-4b1d-af07-30685a4498d0\") " pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.604657 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.604641    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kfh5\" (UniqueName: \"kubernetes.io/projected/2851d0a8-1480-444d-bc7e-ab21239553df-kube-api-access-8kfh5\") pod \"ovnkube-node-8dsbz\" (UID: \"2851d0a8-1480-444d-bc7e-ab21239553df\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.604961 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.604944    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzr4x\" (UniqueName: \"kubernetes.io/projected/e1016a67-9fd8-42c1-a6ac-b7220db66003-kube-api-access-vzr4x\") pod \"aws-ebs-csi-driver-node-5s7gq\" (UID: \"e1016a67-9fd8-42c1-a6ac-b7220db66003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.698648 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.698614    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-68fww"
Jul 20 09:57:08.709042 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.709010    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcd6077_e7cf_4e17_829b_514131357a1a.slice/crio-2182e48875bd93f27a8e257896bf7c476caaa3b4341ed25cc104864109870d98 WatchSource:0}: Error finding container 2182e48875bd93f27a8e257896bf7c476caaa3b4341ed25cc104864109870d98: Status 404 returned error can't find the container with id 2182e48875bd93f27a8e257896bf7c476caaa3b4341ed25cc104864109870d98
Jul 20 09:57:08.716486 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.716465    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tfh95"
Jul 20 09:57:08.725710 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.725684    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0274aa1_d4b8_463c_8f61_d888fe04e631.slice/crio-8813f672859b29f33d62de32d2b0aa55b23e46e3d0f35fa03796dd4d4deec857 WatchSource:0}: Error finding container 8813f672859b29f33d62de32d2b0aa55b23e46e3d0f35fa03796dd4d4deec857: Status 404 returned error can't find the container with id 8813f672859b29f33d62de32d2b0aa55b23e46e3d0f35fa03796dd4d4deec857
Jul 20 09:57:08.738177 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.738152    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kpskk"
Jul 20 09:57:08.747841 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.747807    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75520ae_fef6_42fe_a019_59a2f6b04a24.slice/crio-e1827f0e3af094e25af98bb6d694a565c9f06f20fe6aaf8756f445855fdb742b WatchSource:0}: Error finding container e1827f0e3af094e25af98bb6d694a565c9f06f20fe6aaf8756f445855fdb742b: Status 404 returned error can't find the container with id e1827f0e3af094e25af98bb6d694a565c9f06f20fe6aaf8756f445855fdb742b
Jul 20 09:57:08.749037 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.749022    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cm5ps"
Jul 20 09:57:08.757391 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.757360    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58dc4f4e_0a78_4b1d_af07_30685a4498d0.slice/crio-9798f8cd7a7b5ff0ab7a6e65ce9535bb02f5b06251a1280371443f37ec614ca2 WatchSource:0}: Error finding container 9798f8cd7a7b5ff0ab7a6e65ce9535bb02f5b06251a1280371443f37ec614ca2: Status 404 returned error can't find the container with id 9798f8cd7a7b5ff0ab7a6e65ce9535bb02f5b06251a1280371443f37ec614ca2
Jul 20 09:57:08.765484 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.765458    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq"
Jul 20 09:57:08.774806 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.774766    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1016a67_9fd8_42c1_a6ac_b7220db66003.slice/crio-1d4838da4a0d6650d688e0d2d0aeb1aba2e6edd7d6d054feb04afb63b53bba73 WatchSource:0}: Error finding container 1d4838da4a0d6650d688e0d2d0aeb1aba2e6edd7d6d054feb04afb63b53bba73: Status 404 returned error can't find the container with id 1d4838da4a0d6650d688e0d2d0aeb1aba2e6edd7d6d054feb04afb63b53bba73
Jul 20 09:57:08.778251 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.778234    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xzt4t"
Jul 20 09:57:08.786619 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.786589    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cnzgt"
Jul 20 09:57:08.786901 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.786868    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba89306_92d2_4c9f_a577_d992c350a925.slice/crio-6a265e5a31d41e5ef73988d1aee40ecd2273c48fa304d98b88c2cad350e24d4a WatchSource:0}: Error finding container 6a265e5a31d41e5ef73988d1aee40ecd2273c48fa304d98b88c2cad350e24d4a: Status 404 returned error can't find the container with id 6a265e5a31d41e5ef73988d1aee40ecd2273c48fa304d98b88c2cad350e24d4a
Jul 20 09:57:08.795036 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.795011    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db3a8a2_e368_4a2a_aae2_4ed24d9e4eac.slice/crio-9b72ada9af3e17ddafe3af6e845e6f3f8d723a2defbcb41826db0f1547facb4e WatchSource:0}: Error finding container 9b72ada9af3e17ddafe3af6e845e6f3f8d723a2defbcb41826db0f1547facb4e: Status 404 returned error can't find the container with id 9b72ada9af3e17ddafe3af6e845e6f3f8d723a2defbcb41826db0f1547facb4e
Jul 20 09:57:08.803035 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.803006    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:08.809221 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:08.809192    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:08.826630 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.826587    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2851d0a8_1480_444d_bc7e_ab21239553df.slice/crio-394dd68d894fca6de7221f699cf1008e4b7ab9391b190d9d0c6f086050aecddd WatchSource:0}: Error finding container 394dd68d894fca6de7221f699cf1008e4b7ab9391b190d9d0c6f086050aecddd: Status 404 returned error can't find the container with id 394dd68d894fca6de7221f699cf1008e4b7ab9391b190d9d0c6f086050aecddd
Jul 20 09:57:08.827009 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.826980    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bee537_c2d5_4977_a3fd_b8d6cc021900.slice/crio-67ec43f185932437699ffe71a550db7e3a7540342dca2134bf13908ec6016b21 WatchSource:0}: Error finding container 67ec43f185932437699ffe71a550db7e3a7540342dca2134bf13908ec6016b21: Status 404 returned error can't find the container with id 67ec43f185932437699ffe71a550db7e3a7540342dca2134bf13908ec6016b21
Jul 20 09:57:08.827758 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:08.827714    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2adbd0e228b3393b7ea7327f58790b.slice/crio-86402255afd8a1a31e78f688db87a8d1d7723a2664638ffab19db70d7aecdd36 WatchSource:0}: Error finding container 86402255afd8a1a31e78f688db87a8d1d7723a2664638ffab19db70d7aecdd36: Status 404 returned error can't find the container with id 86402255afd8a1a31e78f688db87a8d1d7723a2664638ffab19db70d7aecdd36
Jul 20 09:57:09.088327 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.088211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:09.088327 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.088303    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:09.088554 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:09.088395    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:09.088554 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:09.088461    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:09.088554 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:09.088489    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs podName:ed36d59b-3fc9-43ea-9506-f28d6955b7a3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:10.088464258 +0000 UTC m=+3.139393606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs") pod "network-metrics-daemon-p24w6" (UID: "ed36d59b-3fc9-43ea-9506-f28d6955b7a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:09.088554 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:09.088525    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret podName:3796601d-5123-4213-8c2d-453d6212d472 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:10.088506779 +0000 UTC m=+3.139436135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret") pod "global-pull-secret-syncer-5r7zg" (UID: "3796601d-5123-4213-8c2d-453d6212d472") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:09.188934 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.188889    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9wb\" (UniqueName: \"kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb\") pod \"network-check-target-q7zvj\" (UID: \"1d7b7476-9151-41f4-a037-54a40c87cdc4\") " pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:09.189141 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:09.189072    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:09.189141 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:09.189102    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:09.189141 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:09.189117    2555 projected.go:196] Error preparing data for projected volume kube-api-access-mx9wb for pod openshift-network-diagnostics/network-check-target-q7zvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:09.189350 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:09.189192    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb podName:1d7b7476-9151-41f4-a037-54a40c87cdc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:10.18917096 +0000 UTC m=+3.240100297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx9wb" (UniqueName: "kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb") pod "network-check-target-q7zvj" (UID: "1d7b7476-9151-41f4-a037-54a40c87cdc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:09.423422 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.423220    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:52:08 +0000 UTC" deadline="2028-02-08 07:31:00.714598526 +0000 UTC"
Jul 20 09:57:09.423422 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.423303    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13629h33m51.291300619s"
Jul 20 09:57:09.451816 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.451560    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:09.489523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.488539    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:09.489523 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:09.488680    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:09.489523 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.489143    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:09.489523 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:09.489284    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:09.501501 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.501371    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kpskk" event={"ID":"b75520ae-fef6-42fe-a019-59a2f6b04a24","Type":"ContainerStarted","Data":"e1827f0e3af094e25af98bb6d694a565c9f06f20fe6aaf8756f445855fdb742b"}
Jul 20 09:57:09.506124 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.506056    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal" event={"ID":"bf485a96b13d861618003dd2b5d77147","Type":"ContainerStarted","Data":"35102e7b071786b3292ebc8f3fa7193d2cef5cb94fe6e1b958f3c05dc1fde4ed"}
Jul 20 09:57:09.510262 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.510225    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cnzgt" event={"ID":"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac","Type":"ContainerStarted","Data":"9b72ada9af3e17ddafe3af6e845e6f3f8d723a2defbcb41826db0f1547facb4e"}
Jul 20 09:57:09.528365 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.523266    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tfh95" event={"ID":"a0274aa1-d4b8-463c-8f61-d888fe04e631","Type":"ContainerStarted","Data":"8813f672859b29f33d62de32d2b0aa55b23e46e3d0f35fa03796dd4d4deec857"}
Jul 20 09:57:09.531797 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.531752    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68fww" event={"ID":"bfcd6077-e7cf-4e17-829b-514131357a1a","Type":"ContainerStarted","Data":"2182e48875bd93f27a8e257896bf7c476caaa3b4341ed25cc104864109870d98"}
Jul 20 09:57:09.552863 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.552729    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal" event={"ID":"ba2adbd0e228b3393b7ea7327f58790b","Type":"ContainerStarted","Data":"86402255afd8a1a31e78f688db87a8d1d7723a2664638ffab19db70d7aecdd36"}
Jul 20 09:57:09.558537 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.558492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c2gwd" event={"ID":"d8bee537-c2d5-4977-a3fd-b8d6cc021900","Type":"ContainerStarted","Data":"67ec43f185932437699ffe71a550db7e3a7540342dca2134bf13908ec6016b21"}
Jul 20 09:57:09.565371 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.565325    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" event={"ID":"2851d0a8-1480-444d-bc7e-ab21239553df","Type":"ContainerStarted","Data":"394dd68d894fca6de7221f699cf1008e4b7ab9391b190d9d0c6f086050aecddd"}
Jul 20 09:57:09.572088 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.569083    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xzt4t" event={"ID":"cba89306-92d2-4c9f-a577-d992c350a925","Type":"ContainerStarted","Data":"6a265e5a31d41e5ef73988d1aee40ecd2273c48fa304d98b88c2cad350e24d4a"}
Jul 20 09:57:09.572088 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.571170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq" event={"ID":"e1016a67-9fd8-42c1-a6ac-b7220db66003","Type":"ContainerStarted","Data":"1d4838da4a0d6650d688e0d2d0aeb1aba2e6edd7d6d054feb04afb63b53bba73"}
Jul 20 09:57:09.574356 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.572607    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cm5ps" event={"ID":"58dc4f4e-0a78-4b1d-af07-30685a4498d0","Type":"ContainerStarted","Data":"9798f8cd7a7b5ff0ab7a6e65ce9535bb02f5b06251a1280371443f37ec614ca2"}
Jul 20 09:57:09.841714 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:09.841068    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:10.098174 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:10.098081    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:10.098374 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:10.098196    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:10.098374 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:10.098360    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:10.098502 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:10.098433    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs podName:ed36d59b-3fc9-43ea-9506-f28d6955b7a3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:12.098411562 +0000 UTC m=+5.149340900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs") pod "network-metrics-daemon-p24w6" (UID: "ed36d59b-3fc9-43ea-9506-f28d6955b7a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:10.099071 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:10.098859    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:10.099071 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:10.098917    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret podName:3796601d-5123-4213-8c2d-453d6212d472 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:12.098900421 +0000 UTC m=+5.149829761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret") pod "global-pull-secret-syncer-5r7zg" (UID: "3796601d-5123-4213-8c2d-453d6212d472") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:10.200296 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:10.199581    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9wb\" (UniqueName: \"kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb\") pod \"network-check-target-q7zvj\" (UID: \"1d7b7476-9151-41f4-a037-54a40c87cdc4\") " pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:10.200296 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:10.199831    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:10.200296 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:10.199854    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:10.200296 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:10.199867    2555 projected.go:196] Error preparing data for projected volume kube-api-access-mx9wb for pod openshift-network-diagnostics/network-check-target-q7zvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:10.200296 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:10.199929    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb podName:1d7b7476-9151-41f4-a037-54a40c87cdc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:12.199909792 +0000 UTC m=+5.250839130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx9wb" (UniqueName: "kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb") pod "network-check-target-q7zvj" (UID: "1d7b7476-9151-41f4-a037-54a40c87cdc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:10.423665 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:10.423565    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:52:08 +0000 UTC" deadline="2028-01-12 22:20:05.869194492 +0000 UTC"
Jul 20 09:57:10.423665 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:10.423609    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12996h22m55.445591392s"
Jul 20 09:57:10.486381 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:10.486245    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:10.486585 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:10.486418    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:10.741440 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:10.741343    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:11.487132 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:11.486865    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:11.487132 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:11.487010    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:11.487679 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:11.487497    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:11.487679 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:11.487649    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:12.116941 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:12.116900    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:12.117130 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:12.117010    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:12.117195 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:12.117148    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:12.117270 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:12.117210    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs podName:ed36d59b-3fc9-43ea-9506-f28d6955b7a3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:16.117192298 +0000 UTC m=+9.168121633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs") pod "network-metrics-daemon-p24w6" (UID: "ed36d59b-3fc9-43ea-9506-f28d6955b7a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:12.117660 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:12.117637    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:12.117762 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:12.117703    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret podName:3796601d-5123-4213-8c2d-453d6212d472 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:16.117684312 +0000 UTC m=+9.168613648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret") pod "global-pull-secret-syncer-5r7zg" (UID: "3796601d-5123-4213-8c2d-453d6212d472") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:12.218901 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:12.218135    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9wb\" (UniqueName: \"kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb\") pod \"network-check-target-q7zvj\" (UID: \"1d7b7476-9151-41f4-a037-54a40c87cdc4\") " pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:12.218901 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:12.218361    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:12.218901 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:12.218387    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:12.218901 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:12.218400    2555 projected.go:196] Error preparing data for projected volume kube-api-access-mx9wb for pod openshift-network-diagnostics/network-check-target-q7zvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:12.218901 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:12.218461    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb podName:1d7b7476-9151-41f4-a037-54a40c87cdc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:16.218440787 +0000 UTC m=+9.269370159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx9wb" (UniqueName: "kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb") pod "network-check-target-q7zvj" (UID: "1d7b7476-9151-41f4-a037-54a40c87cdc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:12.485930 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:12.485848    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:12.486087 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:12.486007    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:13.486347 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:13.486308    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:13.486937 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:13.486453    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:13.486937 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:13.486826    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:13.487059 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:13.486965    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:14.486023 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:14.485973    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:14.486222 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:14.486118    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:15.486624 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:15.486587    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:15.487181 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:15.486740    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:15.487181 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:15.486793    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:15.487181 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:15.486920    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:16.158136 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:16.157270    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:16.158136 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:16.157337    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:16.158136 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:16.157536    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:16.158136 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:16.157633    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret podName:3796601d-5123-4213-8c2d-453d6212d472 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:24.157589267 +0000 UTC m=+17.208518607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret") pod "global-pull-secret-syncer-5r7zg" (UID: "3796601d-5123-4213-8c2d-453d6212d472") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:16.158136 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:16.158058    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:16.158136 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:16.158113    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs podName:ed36d59b-3fc9-43ea-9506-f28d6955b7a3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:24.1580956 +0000 UTC m=+17.209024936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs") pod "network-metrics-daemon-p24w6" (UID: "ed36d59b-3fc9-43ea-9506-f28d6955b7a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:16.258200 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:16.258066    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9wb\" (UniqueName: \"kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb\") pod \"network-check-target-q7zvj\" (UID: \"1d7b7476-9151-41f4-a037-54a40c87cdc4\") " pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:16.258376 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:16.258267    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:16.258376 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:16.258296    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:16.258376 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:16.258309    2555 projected.go:196] Error preparing data for projected volume kube-api-access-mx9wb for pod openshift-network-diagnostics/network-check-target-q7zvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:16.258546 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:16.258376    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb podName:1d7b7476-9151-41f4-a037-54a40c87cdc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:24.258357768 +0000 UTC m=+17.309287105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx9wb" (UniqueName: "kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb") pod "network-check-target-q7zvj" (UID: "1d7b7476-9151-41f4-a037-54a40c87cdc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:16.485551 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:16.485511    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:16.485729 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:16.485650    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:17.487051 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:17.486996    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:17.487530 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:17.487144    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:17.487621 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:17.487581    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:17.487721 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:17.487696    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:18.486594 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:18.486548    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:18.486770 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:18.486702    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:19.486165 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:19.486128    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:19.486638 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:19.486141    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:19.486638 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:19.486310    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:19.486638 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:19.486362    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:20.485646 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:20.485608    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:20.485831 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:20.485720    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:21.486631 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:21.486541    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:21.487066 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:21.486550    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:21.487066 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:21.486689    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:21.487066 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:21.486738    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:22.486341 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:22.486300    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:22.486532 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:22.486439    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:23.486408 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:23.486370    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:23.486893 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:23.486370    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:23.486893 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:23.486517    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:23.486893 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:23.486618    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:24.222053 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:24.222008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:24.222323 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:24.222074    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:24.222323 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:24.222182    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:24.222323 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:24.222263    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs podName:ed36d59b-3fc9-43ea-9506-f28d6955b7a3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:40.222231332 +0000 UTC m=+33.273160668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs") pod "network-metrics-daemon-p24w6" (UID: "ed36d59b-3fc9-43ea-9506-f28d6955b7a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:24.222323 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:24.222182    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:24.222533 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:24.222376    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret podName:3796601d-5123-4213-8c2d-453d6212d472 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:40.222346969 +0000 UTC m=+33.273276317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret") pod "global-pull-secret-syncer-5r7zg" (UID: "3796601d-5123-4213-8c2d-453d6212d472") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:24.322454 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:24.322414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9wb\" (UniqueName: \"kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb\") pod \"network-check-target-q7zvj\" (UID: \"1d7b7476-9151-41f4-a037-54a40c87cdc4\") " pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:24.322647 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:24.322611    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:24.322647 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:24.322635    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:24.322647 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:24.322645    2555 projected.go:196] Error preparing data for projected volume kube-api-access-mx9wb for pod openshift-network-diagnostics/network-check-target-q7zvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:24.322771 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:24.322699    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb podName:1d7b7476-9151-41f4-a037-54a40c87cdc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:40.322685152 +0000 UTC m=+33.373614485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx9wb" (UniqueName: "kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb") pod "network-check-target-q7zvj" (UID: "1d7b7476-9151-41f4-a037-54a40c87cdc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:24.485975 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:24.485886    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:24.486156 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:24.486019    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:25.486375 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:25.486339    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:25.486844 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:25.486340    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:25.486844 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:25.486488    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:25.486844 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:25.486536    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:26.486176 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:26.486138    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:26.486349 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:26.486271    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:27.487114 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:27.487074    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:27.487603 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:27.487174    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:27.487603 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:27.487189    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:27.487603 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:27.487341    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:28.485947 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:28.485910    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:28.486126 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:28.486046    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:29.485974 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:29.485821    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:29.486407 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:29.486074    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:29.486407 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:29.485891    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:29.486407 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:29.486238    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:29.623542 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:29.623499    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xzt4t" event={"ID":"cba89306-92d2-4c9f-a577-d992c350a925","Type":"ContainerStarted","Data":"513e5cd5b079c60f98afcc23794b6d7170a6c168850e3faa2a7e1837df917280"}
Jul 20 09:57:29.625058 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:29.625026    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cnzgt" event={"ID":"2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac","Type":"ContainerStarted","Data":"cbef5a1aa8b9215b8331610fecaf1ef9752150a881eaf3c27e26bcfd25888d46"}
Jul 20 09:57:29.628115 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:29.628090    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal" event={"ID":"ba2adbd0e228b3393b7ea7327f58790b","Type":"ContainerStarted","Data":"07e86ecfdcba47f278741779dd38674d6db288bb0860481619c5768bb2ac6d46"}
Jul 20 09:57:29.685467 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:29.685367    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xzt4t" podStartSLOduration=2.114586015 podStartE2EDuration="22.685349667s" totalImagesPullingTime="20.570763652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:29.647925097 +0000 UTC m=+22.698854451" watchObservedRunningTime="2026-07-20 09:57:29.685349667 +0000 UTC m=+22.736279024"
Jul 20 09:57:29.685746 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:29.685695    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cnzgt" podStartSLOduration=2.12623629 podStartE2EDuration="22.68568368s" totalImagesPullingTime="20.55944739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:29.684730841 +0000 UTC m=+22.735660230" watchObservedRunningTime="2026-07-20 09:57:29.68568368 +0000 UTC m=+22.736613047"
Jul 20 09:57:29.707875 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:29.707809    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-35.ec2.internal" podStartSLOduration=21.707784512 podStartE2EDuration="21.707784512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:29.707131277 +0000 UTC m=+22.758060631" watchObservedRunningTime="2026-07-20 09:57:29.707784512 +0000 UTC m=+22.758713868"
Jul 20 09:57:30.486651 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.486478    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:30.487396 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:30.486747    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:30.631351 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.631312    2555 generic.go:356] "Generic (PLEG): container finished" podID="bf485a96b13d861618003dd2b5d77147" containerID="bfff4e594d92a03d267a4b4f97e0e13c1e298ecd3e358db2ab8ab3ce7eb62670" exitCode=0
Jul 20 09:57:30.631549 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.631403    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal" event={"ID":"bf485a96b13d861618003dd2b5d77147","Type":"ContainerDied","Data":"bfff4e594d92a03d267a4b4f97e0e13c1e298ecd3e358db2ab8ab3ce7eb62670"}
Jul 20 09:57:30.632917 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.632882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tfh95" event={"ID":"a0274aa1-d4b8-463c-8f61-d888fe04e631","Type":"ContainerStarted","Data":"abbc592abada7d6574ea7f6fbed60e76cf09b8bcb0e7c719b8e527bc3a294ebb"}
Jul 20 09:57:30.634444 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.634421    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfcd6077-e7cf-4e17-829b-514131357a1a" containerID="531f59c847441385a0c7576eab37da3749636a199fd8a8e5a2b948421f326731" exitCode=0
Jul 20 09:57:30.634542 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.634494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68fww" event={"ID":"bfcd6077-e7cf-4e17-829b-514131357a1a","Type":"ContainerDied","Data":"531f59c847441385a0c7576eab37da3749636a199fd8a8e5a2b948421f326731"}
Jul 20 09:57:30.635995 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.635966    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c2gwd" event={"ID":"d8bee537-c2d5-4977-a3fd-b8d6cc021900","Type":"ContainerStarted","Data":"b9caabe997dd71e6760647f61b01106a0edd56f975ab812e05baa6f30f9363a4"}
Jul 20 09:57:30.638781 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.638763    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 09:57:30.639114 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.639095    2555 generic.go:356] "Generic (PLEG): container finished" podID="2851d0a8-1480-444d-bc7e-ab21239553df" containerID="0beb8391f53b8c9dabddb5b93885d52cb87ac9ff71e3dac73e2ee6488c60f230" exitCode=1
Jul 20 09:57:30.639185 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.639160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" event={"ID":"2851d0a8-1480-444d-bc7e-ab21239553df","Type":"ContainerStarted","Data":"3de6f383270a7c260b0406f9deabd299edae2afaa316dc7a8d6795c59c6d9208"}
Jul 20 09:57:30.639235 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.639221    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" event={"ID":"2851d0a8-1480-444d-bc7e-ab21239553df","Type":"ContainerStarted","Data":"5fa6916dd0c78906f5cf405c4a524ffc1a290e8cc2aaa3c24efeba0a61fd1e25"}
Jul 20 09:57:30.639315 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.639287    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" event={"ID":"2851d0a8-1480-444d-bc7e-ab21239553df","Type":"ContainerStarted","Data":"adc6e6eca1cf18591374b36930d77fc7a6eb2a3f6ca30b56e106d4ed5ba50c3b"}
Jul 20 09:57:30.639369 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.639326    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" event={"ID":"2851d0a8-1480-444d-bc7e-ab21239553df","Type":"ContainerStarted","Data":"d59b8a1d478621e0de5b8c931b7061663c0d71c7598fb00b5c81fbf3728ef826"}
Jul 20 09:57:30.639369 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.639367    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" event={"ID":"2851d0a8-1480-444d-bc7e-ab21239553df","Type":"ContainerDied","Data":"0beb8391f53b8c9dabddb5b93885d52cb87ac9ff71e3dac73e2ee6488c60f230"}
Jul 20 09:57:30.639486 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.639471    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" event={"ID":"2851d0a8-1480-444d-bc7e-ab21239553df","Type":"ContainerStarted","Data":"aadc13ed32a03f0c1a9f2aaf05acff3659ca23e46ad6d8ba521fdbc94fb58cd5"}
Jul 20 09:57:30.640794 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.640771    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq" event={"ID":"e1016a67-9fd8-42c1-a6ac-b7220db66003","Type":"ContainerStarted","Data":"29055783cc026851488b404389b8bd2efbda2592c7fd6fa2b22b29145b2c7c4c"}
Jul 20 09:57:30.642133 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.642115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cm5ps" event={"ID":"58dc4f4e-0a78-4b1d-af07-30685a4498d0","Type":"ContainerStarted","Data":"b056fd94866d99f18946d52bf67dc43519d32c1815e42c7db8c26ace281ff68e"}
Jul 20 09:57:30.700941 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.700712    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c2gwd" podStartSLOduration=3.186364185 podStartE2EDuration="23.700693056s" totalImagesPullingTime="20.514328871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:30.674125396 +0000 UTC m=+23.725054752" watchObservedRunningTime="2026-07-20 09:57:30.700693056 +0000 UTC m=+23.751622410"
Jul 20 09:57:30.723054 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.723000    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tfh95" podStartSLOduration=3.127833763 podStartE2EDuration="23.722982127s" totalImagesPullingTime="20.595148364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:30.722561128 +0000 UTC m=+23.773490484" watchObservedRunningTime="2026-07-20 09:57:30.722982127 +0000 UTC m=+23.773911481"
Jul 20 09:57:30.741160 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.741110    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cm5ps" podStartSLOduration=3.156835009 podStartE2EDuration="23.741092065s" totalImagesPullingTime="20.584257056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:30.740974397 +0000 UTC m=+23.791903764" watchObservedRunningTime="2026-07-20 09:57:30.741092065 +0000 UTC m=+23.792021419"
Jul 20 09:57:30.899199 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:30.899160    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:31.486477 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:31.486442    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:31.486672 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:31.486586    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:31.486672 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:31.486612    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:31.487232 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:31.486743    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:31.510388 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:31.510353    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:31.610713 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:31.610673    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:31.646077 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:31.646040    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kpskk" event={"ID":"b75520ae-fef6-42fe-a019-59a2f6b04a24","Type":"ContainerStarted","Data":"bb1b9056ee49503e19b1553019f83cd808aff5ccfadcd20ad5d0a0c32e88f150"}
Jul 20 09:57:31.648152 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:31.648122    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal" event={"ID":"bf485a96b13d861618003dd2b5d77147","Type":"ContainerStarted","Data":"21cf38badf1c5b0c24f11d68bee6b54adc2dead8444566207051c80f71d8d693"}
Jul 20 09:57:31.664013 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:31.663958    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kpskk" podStartSLOduration=4.051237604 podStartE2EDuration="24.663939711s" totalImagesPullingTime="20.612702107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:31.663898119 +0000 UTC m=+24.714827475" watchObservedRunningTime="2026-07-20 09:57:31.663939711 +0000 UTC m=+24.714869069"
Jul 20 09:57:31.769759 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:31.769719    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:31.858059 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:31.858025    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 20 09:57:31.859853 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:31.859831    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:32.124303 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.124268    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:32.225976 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.225940    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:32.317734 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.317699    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:32.318615 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.318592    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:32.338202 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.338143    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-35.ec2.internal" podStartSLOduration=24.338121597 podStartE2EDuration="24.338121597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:31.681854534 +0000 UTC m=+24.732783887" watchObservedRunningTime="2026-07-20 09:57:32.338121597 +0000 UTC m=+25.389050953"
Jul 20 09:57:32.446958 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.446828    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T09:57:31.858043567Z","UUID":"218665e1-1580-4283-b835-37931fa0df89","Handler":null,"Name":"","Endpoint":""}
Jul 20 09:57:32.449075 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.449051    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 20 09:57:32.449211 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.449099    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 20 09:57:32.486141 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.486113    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:32.486367 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:32.486230    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:32.560294 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.560228    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:32.619761 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.619566    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:32.654895 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.654812    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 09:57:32.655491 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.655467    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" event={"ID":"2851d0a8-1480-444d-bc7e-ab21239553df","Type":"ContainerStarted","Data":"f7bf48cc7b7dd967a90fb8734779a235e0c69cf9ad2f7cf33db980ff4d03db54"}
Jul 20 09:57:32.657695 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.657654    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq" event={"ID":"e1016a67-9fd8-42c1-a6ac-b7220db66003","Type":"ContainerStarted","Data":"67b0364b33791fc562f2a46b2093c54ffcd1689dfba9e2773716567e0aa5be01"}
Jul 20 09:57:32.658103 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.658075    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:32.658582 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.658559    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c2gwd"
Jul 20 09:57:32.722593 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:32.722445    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:33.489459 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:33.489420    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:33.489710 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:33.489420    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:33.489710 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:33.489547    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:33.489710 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:33.489622    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:33.662423 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:33.662377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq" event={"ID":"e1016a67-9fd8-42c1-a6ac-b7220db66003","Type":"ContainerStarted","Data":"7f24c7e255c2ad011be84b1187b5587fe579b1f38417c761b996f0b7cc39d1d4"}
Jul 20 09:57:34.485722 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:34.485684    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:34.485908 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:34.485825    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:35.486709 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:35.486525    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:35.487372 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:35.486550    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:35.487372 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:35.486802    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:35.487372 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:35.486885    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:35.670106 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:35.670078    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 09:57:35.670498 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:35.670469    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" event={"ID":"2851d0a8-1480-444d-bc7e-ab21239553df","Type":"ContainerStarted","Data":"af5d8e4ca6fe9e6c6144145c26d2a2a0c12f0b1bf908d8d8fa11f02b7ad54129"}
Jul 20 09:57:35.670923 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:35.670897    2555 scope.go:117] "RemoveContainer" containerID="0beb8391f53b8c9dabddb5b93885d52cb87ac9ff71e3dac73e2ee6488c60f230"
Jul 20 09:57:35.672371 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:35.672336    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfcd6077-e7cf-4e17-829b-514131357a1a" containerID="28d68343ea699bb7be531e75d034547da7e83a29c836f9f51f3b4f6225a203a8" exitCode=0
Jul 20 09:57:35.672502 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:35.672383    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68fww" event={"ID":"bfcd6077-e7cf-4e17-829b-514131357a1a","Type":"ContainerDied","Data":"28d68343ea699bb7be531e75d034547da7e83a29c836f9f51f3b4f6225a203a8"}
Jul 20 09:57:35.705408 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:35.705361    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5s7gq" podStartSLOduration=5.755095508 podStartE2EDuration="28.705344013s" totalImagesPullingTime="22.950248505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:33.68803699 +0000 UTC m=+26.738966538" watchObservedRunningTime="2026-07-20 09:57:35.705344013 +0000 UTC m=+28.756273406"
Jul 20 09:57:36.485908 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:36.485867    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:36.486119 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:36.485972    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:36.678712 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:36.678687    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 09:57:36.679427 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:36.679046    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" event={"ID":"2851d0a8-1480-444d-bc7e-ab21239553df","Type":"ContainerStarted","Data":"1710aecbff5da898a5f7e7c4f0efa0f810e887d640d9d15248ebeb28441bddc9"}
Jul 20 09:57:36.679631 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:36.679437    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:36.679631 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:36.679519    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:36.679631 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:36.679569    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:36.695652 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:36.695624    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:36.695793 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:36.695735    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:57:36.714808 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:36.714756    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz" podStartSLOduration=8.991324557 podStartE2EDuration="29.714741502s" totalImagesPullingTime="20.723416945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:36.713387239 +0000 UTC m=+29.764316595" watchObservedRunningTime="2026-07-20 09:57:36.714741502 +0000 UTC m=+29.765670856"
Jul 20 09:57:37.029488 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.029384    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5r7zg"]
Jul 20 09:57:37.029697 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.029506    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:37.029697 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:37.029604    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:37.032602 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.032546    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q7zvj"]
Jul 20 09:57:37.032762 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.032670    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:37.032816 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:37.032766    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:37.033408 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.033384    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p24w6"]
Jul 20 09:57:37.033511 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.033487    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:37.033584 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:37.033567    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:37.511276 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.511219    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:37.594587 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.594552    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:37.683581 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.683548    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfcd6077-e7cf-4e17-829b-514131357a1a" containerID="ce6c4b8b1f7bfa13372e7ef499071e32403c6e9e44d2a78039b827684e157466" exitCode=0
Jul 20 09:57:37.683950 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.683630    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68fww" event={"ID":"bfcd6077-e7cf-4e17-829b-514131357a1a","Type":"ContainerDied","Data":"ce6c4b8b1f7bfa13372e7ef499071e32403c6e9e44d2a78039b827684e157466"}
Jul 20 09:57:37.730994 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.730954    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:37.804759 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:37.804653    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:38.486111 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:38.486074    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:38.486301 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:38.486126    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:38.486301 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:38.486074    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:38.486301 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:38.486200    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:38.486301 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:38.486292    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:38.486436 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:38.486367    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:38.570117 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:38.570082    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:38.649235 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:38.649185    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:38.689116 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:38.689082    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68fww" event={"ID":"bfcd6077-e7cf-4e17-829b-514131357a1a","Type":"ContainerStarted","Data":"d96ba46e87cb96dc17705fc6a6784afb16812d36c1d2ef330eb4a5dafab985b0"}
Jul 20 09:57:39.453066 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:39.453028    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:39.693732 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:39.693698    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfcd6077-e7cf-4e17-829b-514131357a1a" containerID="d96ba46e87cb96dc17705fc6a6784afb16812d36c1d2ef330eb4a5dafab985b0" exitCode=0
Jul 20 09:57:39.694115 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:39.693751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68fww" event={"ID":"bfcd6077-e7cf-4e17-829b-514131357a1a","Type":"ContainerDied","Data":"d96ba46e87cb96dc17705fc6a6784afb16812d36c1d2ef330eb4a5dafab985b0"}
Jul 20 09:57:40.242606 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:40.242571    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:40.242606 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:40.242616    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:40.242865 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.242733    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:40.242865 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.242733    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:40.242865 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.242805    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret podName:3796601d-5123-4213-8c2d-453d6212d472 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:12.242786222 +0000 UTC m=+65.293715556 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret") pod "global-pull-secret-syncer-5r7zg" (UID: "3796601d-5123-4213-8c2d-453d6212d472") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:40.242865 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.242825    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs podName:ed36d59b-3fc9-43ea-9506-f28d6955b7a3 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:12.242812946 +0000 UTC m=+65.293742293 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs") pod "network-metrics-daemon-p24w6" (UID: "ed36d59b-3fc9-43ea-9506-f28d6955b7a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:40.343126 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:40.343088    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9wb\" (UniqueName: \"kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb\") pod \"network-check-target-q7zvj\" (UID: \"1d7b7476-9151-41f4-a037-54a40c87cdc4\") " pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:40.343361 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.343266    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:40.343361 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.343288    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:40.343361 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.343300    2555 projected.go:196] Error preparing data for projected volume kube-api-access-mx9wb for pod openshift-network-diagnostics/network-check-target-q7zvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:40.343534 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.343368    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb podName:1d7b7476-9151-41f4-a037-54a40c87cdc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:12.343349717 +0000 UTC m=+65.394279053 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx9wb" (UniqueName: "kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb") pod "network-check-target-q7zvj" (UID: "1d7b7476-9151-41f4-a037-54a40c87cdc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:40.485482 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:40.485450    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:40.485482 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:40.485473    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:40.485482 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:40.485473    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:40.485766 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.485584    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:40.485766 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.485665    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:40.485766 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:40.485741    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:42.485989 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.485739    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:42.486439 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.485757    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:42.486439 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:42.486038    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p24w6" podUID="ed36d59b-3fc9-43ea-9506-f28d6955b7a3"
Jul 20 09:57:42.486439 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.485757    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:42.486439 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:42.486128    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5r7zg" podUID="3796601d-5123-4213-8c2d-453d6212d472"
Jul 20 09:57:42.486439 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:42.486179    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7zvj" podUID="1d7b7476-9151-41f4-a037-54a40c87cdc4"
Jul 20 09:57:42.764231 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.764144    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-35.ec2.internal" event="NodeReady"
Jul 20 09:57:42.764410 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.764323    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 20 09:57:42.812082 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.812029    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mlj89"]
Jul 20 09:57:42.816213 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.816182    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jgts6"]
Jul 20 09:57:42.816400 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.816346    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:42.819864 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.819781    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 20 09:57:42.819999 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.819877    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l5b2l\""
Jul 20 09:57:42.819999 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.819827    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jgts6"
Jul 20 09:57:42.820171 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.820149    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 20 09:57:42.822475 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.822454    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 20 09:57:42.823123 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.823094    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 20 09:57:42.826502 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.823614    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-84ttf\""
Jul 20 09:57:42.826502 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.823869    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 20 09:57:42.829404 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.829381    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mlj89"]
Jul 20 09:57:42.830321 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.830295    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jgts6"]
Jul 20 09:57:42.961160 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.961117    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-config-volume\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:42.961361 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.961175    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgvhw\" (UniqueName: \"kubernetes.io/projected/630a07a2-1674-43f8-82d6-08df700b040c-kube-api-access-rgvhw\") pod \"ingress-canary-jgts6\" (UID: \"630a07a2-1674-43f8-82d6-08df700b040c\") " pod="openshift-ingress-canary/ingress-canary-jgts6"
Jul 20 09:57:42.961361 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.961207    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:42.961361 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.961232    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-tmp-dir\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:42.961361 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.961330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cl2w\" (UniqueName: \"kubernetes.io/projected/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-kube-api-access-6cl2w\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:42.961538 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.961407    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/630a07a2-1674-43f8-82d6-08df700b040c-cert\") pod \"ingress-canary-jgts6\" (UID: \"630a07a2-1674-43f8-82d6-08df700b040c\") " pod="openshift-ingress-canary/ingress-canary-jgts6"
Jul 20 09:57:42.975333 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:42.975294    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:43.036917 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.036810    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:43.062637 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.062590    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-config-volume\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:43.062637 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.062637    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgvhw\" (UniqueName: \"kubernetes.io/projected/630a07a2-1674-43f8-82d6-08df700b040c-kube-api-access-rgvhw\") pod \"ingress-canary-jgts6\" (UID: \"630a07a2-1674-43f8-82d6-08df700b040c\") " pod="openshift-ingress-canary/ingress-canary-jgts6"
Jul 20 09:57:43.062899 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.062660    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:43.062899 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.062679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-tmp-dir\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:43.062899 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.062720    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cl2w\" (UniqueName: \"kubernetes.io/projected/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-kube-api-access-6cl2w\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:43.062899 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.062774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/630a07a2-1674-43f8-82d6-08df700b040c-cert\") pod \"ingress-canary-jgts6\" (UID: \"630a07a2-1674-43f8-82d6-08df700b040c\") " pod="openshift-ingress-canary/ingress-canary-jgts6"
Jul 20 09:57:43.062899 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:43.062776    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:43.062899 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:43.062872    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls podName:c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:43.562851541 +0000 UTC m=+36.613780874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls") pod "dns-default-mlj89" (UID: "c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:43.063172 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.063133    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-tmp-dir\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:43.063347 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.063325    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-config-volume\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:43.067726 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.067697    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/630a07a2-1674-43f8-82d6-08df700b040c-cert\") pod \"ingress-canary-jgts6\" (UID: \"630a07a2-1674-43f8-82d6-08df700b040c\") " pod="openshift-ingress-canary/ingress-canary-jgts6"
Jul 20 09:57:43.073207 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.073177    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgvhw\" (UniqueName: \"kubernetes.io/projected/630a07a2-1674-43f8-82d6-08df700b040c-kube-api-access-rgvhw\") pod \"ingress-canary-jgts6\" (UID: \"630a07a2-1674-43f8-82d6-08df700b040c\") " pod="openshift-ingress-canary/ingress-canary-jgts6"
Jul 20 09:57:43.073570 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.073543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cl2w\" (UniqueName: \"kubernetes.io/projected/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-kube-api-access-6cl2w\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:43.139105 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.139068    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jgts6"
Jul 20 09:57:43.567101 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.567054    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:43.567591 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:43.567193    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:43.567591 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:43.567272    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls podName:c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:44.56723385 +0000 UTC m=+37.618163183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls") pod "dns-default-mlj89" (UID: "c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:43.965321 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:43.965296    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jgts6"]
Jul 20 09:57:43.968485 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:57:43.968454    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630a07a2_1674_43f8_82d6_08df700b040c.slice/crio-72dfed161b740ac2cd4d763bccf1fe593961e26b9c31b4531d4c165d6eb7d66f WatchSource:0}: Error finding container 72dfed161b740ac2cd4d763bccf1fe593961e26b9c31b4531d4c165d6eb7d66f: Status 404 returned error can't find the container with id 72dfed161b740ac2cd4d763bccf1fe593961e26b9c31b4531d4c165d6eb7d66f
Jul 20 09:57:44.485645 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.485601    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:57:44.485848 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.485601    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:57:44.485848 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.485601    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:57:44.489446 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.489410    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8dwl2\""
Jul 20 09:57:44.489597 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.489468    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:57:44.490469 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.490444    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wrt6f\""
Jul 20 09:57:44.490596 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.490481    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:57:44.490596 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.490500    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:57:44.490596 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.490451    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:57:44.573039 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.572999    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:44.573788 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:44.573136    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:44.573788 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:44.573213    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls podName:c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:46.573191289 +0000 UTC m=+39.624120648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls") pod "dns-default-mlj89" (UID: "c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:44.707900 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.707868    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfcd6077-e7cf-4e17-829b-514131357a1a" containerID="5087f61d61c2d180b70fa5e9654424e4715ff66ca86971a796be9887d979d327" exitCode=0
Jul 20 09:57:44.708082 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.707948    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68fww" event={"ID":"bfcd6077-e7cf-4e17-829b-514131357a1a","Type":"ContainerDied","Data":"5087f61d61c2d180b70fa5e9654424e4715ff66ca86971a796be9887d979d327"}
Jul 20 09:57:44.709301 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:44.709262    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jgts6" event={"ID":"630a07a2-1674-43f8-82d6-08df700b040c","Type":"ContainerStarted","Data":"72dfed161b740ac2cd4d763bccf1fe593961e26b9c31b4531d4c165d6eb7d66f"}
Jul 20 09:57:45.719099 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:45.719069    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfcd6077-e7cf-4e17-829b-514131357a1a" containerID="eb9f79c5304d7bd3c2ab3c2ce5c91ad0c6c74313bc68b65e1628c05bf296d7da" exitCode=0
Jul 20 09:57:45.719581 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:45.719122    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68fww" event={"ID":"bfcd6077-e7cf-4e17-829b-514131357a1a","Type":"ContainerDied","Data":"eb9f79c5304d7bd3c2ab3c2ce5c91ad0c6c74313bc68b65e1628c05bf296d7da"}
Jul 20 09:57:46.589474 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:46.589435    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:46.589622 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:46.589583    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:46.589685 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:46.589647    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls podName:c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:50.589630504 +0000 UTC m=+43.640559837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls") pod "dns-default-mlj89" (UID: "c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:46.724941 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:46.724858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68fww" event={"ID":"bfcd6077-e7cf-4e17-829b-514131357a1a","Type":"ContainerStarted","Data":"4d4551003204f22649022ec879db40481c2826f96d5f310f783a880c44de65de"}
Jul 20 09:57:46.726199 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:46.726175    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jgts6" event={"ID":"630a07a2-1674-43f8-82d6-08df700b040c","Type":"ContainerStarted","Data":"c8bc729695771dc8ca8e914ee409d9a7ed852c2c1fd0d99577d0c4adc3a9c648"}
Jul 20 09:57:46.763135 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:46.763073    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-68fww" podStartSLOduration=8.723905622 podStartE2EDuration="39.763054515s" totalImagesPullingTime="31.039148893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:46.762374682 +0000 UTC m=+39.813304036" watchObservedRunningTime="2026-07-20 09:57:46.763054515 +0000 UTC m=+39.813983877"
Jul 20 09:57:46.801132 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:46.801082    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jgts6" podStartSLOduration=2.813369927 podStartE2EDuration="4.801066462s" totalImagesPullingTime="1.987696535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:46.801009275 +0000 UTC m=+39.851938631" watchObservedRunningTime="2026-07-20 09:57:46.801066462 +0000 UTC m=+39.851995814"
Jul 20 09:57:46.924027 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:46.923989    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:47.027308 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:47.027206    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:50.621372 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:50.621320    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:50.621789 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:50.621487    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:50.621789 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:50.621559    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls podName:c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:58.621541543 +0000 UTC m=+51.672470879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls") pod "dns-default-mlj89" (UID: "c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:50.833595 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:50.833551    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:50.926142 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:50.926060    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:53.563136 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:53.563098    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:53.664137 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:53.664100    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:53.722453 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:53.722419    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:54.243833 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:54.243793    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:54.323963 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:54.323934    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:55.979031 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:55.978995    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:56.038491 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:56.038454    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:58.678870 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:57:58.678834    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:57:58.679317 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:58.679000    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:58.679317 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:57:58.679068    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls podName:c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:14.679048884 +0000 UTC m=+67.729978221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls") pod "dns-default-mlj89" (UID: "c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3") : secret "dns-default-metrics-tls" not found
Jul 20 09:58:08.702658 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:08.702618    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dsbz"
Jul 20 09:58:08.866756 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:08.866713    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:08.929952 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:08.929916    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:12.271357 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.271303    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:58:12.271767 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.271412    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:58:12.274290 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.274262    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:58:12.274397 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.274379    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:58:12.282020 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:12.282004    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 20 09:58:12.282073 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:12.282062    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs podName:ed36d59b-3fc9-43ea-9506-f28d6955b7a3 nodeName:}" failed. No retries permitted until 2026-07-20 09:59:16.282045672 +0000 UTC m=+129.332975008 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs") pod "network-metrics-daemon-p24w6" (UID: "ed36d59b-3fc9-43ea-9506-f28d6955b7a3") : secret "metrics-daemon-secret" not found
Jul 20 09:58:12.285182 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.285158    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3796601d-5123-4213-8c2d-453d6212d472-original-pull-secret\") pod \"global-pull-secret-syncer-5r7zg\" (UID: \"3796601d-5123-4213-8c2d-453d6212d472\") " pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:58:12.372010 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.371953    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9wb\" (UniqueName: \"kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb\") pod \"network-check-target-q7zvj\" (UID: \"1d7b7476-9151-41f4-a037-54a40c87cdc4\") " pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:58:12.375345 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.375325    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:58:12.385953 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.385928    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:58:12.395872 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.395842    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx9wb\" (UniqueName: \"kubernetes.io/projected/1d7b7476-9151-41f4-a037-54a40c87cdc4-kube-api-access-mx9wb\") pod \"network-check-target-q7zvj\" (UID: \"1d7b7476-9151-41f4-a037-54a40c87cdc4\") " pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:58:12.397626 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.397604    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5r7zg"
Jul 20 09:58:12.413818 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.413790    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wrt6f\""
Jul 20 09:58:12.421403 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.421379    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:58:12.537104 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.537025    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5r7zg"]
Jul 20 09:58:12.539984 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:58:12.539957    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3796601d_5123_4213_8c2d_453d6212d472.slice/crio-f1f72ebd80d5c2937a278b6e328ca055c3b8b48ee296a19bd6ea08e866c1f012 WatchSource:0}: Error finding container f1f72ebd80d5c2937a278b6e328ca055c3b8b48ee296a19bd6ea08e866c1f012: Status 404 returned error can't find the container with id f1f72ebd80d5c2937a278b6e328ca055c3b8b48ee296a19bd6ea08e866c1f012
Jul 20 09:58:12.553847 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.553820    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q7zvj"]
Jul 20 09:58:12.557066 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:58:12.557035    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7b7476_9151_41f4_a037_54a40c87cdc4.slice/crio-5c8d35b14e9aec5aa93a780f6c1674277a5a5e41b83239a704d28a14e9d539bf WatchSource:0}: Error finding container 5c8d35b14e9aec5aa93a780f6c1674277a5a5e41b83239a704d28a14e9d539bf: Status 404 returned error can't find the container with id 5c8d35b14e9aec5aa93a780f6c1674277a5a5e41b83239a704d28a14e9d539bf
Jul 20 09:58:12.787551 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.787456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q7zvj" event={"ID":"1d7b7476-9151-41f4-a037-54a40c87cdc4","Type":"ContainerStarted","Data":"5c8d35b14e9aec5aa93a780f6c1674277a5a5e41b83239a704d28a14e9d539bf"}
Jul 20 09:58:12.788535 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:12.788509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5r7zg" event={"ID":"3796601d-5123-4213-8c2d-453d6212d472","Type":"ContainerStarted","Data":"f1f72ebd80d5c2937a278b6e328ca055c3b8b48ee296a19bd6ea08e866c1f012"}
Jul 20 09:58:14.691318 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:14.691269    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:58:14.691816 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:14.691445    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:58:14.691816 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:14.691525    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls podName:c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:46.691502084 +0000 UTC m=+99.742431432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls") pod "dns-default-mlj89" (UID: "c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3") : secret "dns-default-metrics-tls" not found
Jul 20 09:58:15.587112 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:15.587062    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:15.646553 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:15.646514    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:16.719190 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.719150    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"]
Jul 20 09:58:16.735630 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.735583    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"]
Jul 20 09:58:16.735808 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.735707    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:16.740864 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.740829    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 20 09:58:16.743624 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.743600    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 20 09:58:16.743787 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.743769    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 20 09:58:16.743879 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.743861    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 20 09:58:16.744188 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.744032    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8656c\""
Jul 20 09:58:16.840187 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.840146    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-67c67595d6-ltx5q"]
Jul 20 09:58:16.857487 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.857456    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:16.866741 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.866713    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 20 09:58:16.867504 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.866721    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 20 09:58:16.867504 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.866995    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 20 09:58:16.867504 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.867070    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 20 09:58:16.867504 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.867215    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 20 09:58:16.867504 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.867375    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 20 09:58:16.867504 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.867440    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jj8cc\""
Jul 20 09:58:16.867968 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.867951    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67c67595d6-ltx5q"]
Jul 20 09:58:16.905726 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.905694    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5bx5\" (UniqueName: \"kubernetes.io/projected/58f197ee-2586-441b-b591-66090ab5dd32-kube-api-access-m5bx5\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:16.905914 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.905753    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:16.905914 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:16.905825    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/58f197ee-2586-441b-b591-66090ab5dd32-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:17.006767 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.006681    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-default-certificate\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.006767 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.006727    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.006767 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.006760    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:17.007024 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.006789    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/58f197ee-2586-441b-b591-66090ab5dd32-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:17.007024 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.006832    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.007024 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.006861    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-stats-auth\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.007024 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.006885    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmm42\" (UniqueName: \"kubernetes.io/projected/17037412-22f7-435b-9b7c-54d02129d8c5-kube-api-access-cmm42\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.007024 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:17.006910    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:17.007024 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.006932    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5bx5\" (UniqueName: \"kubernetes.io/projected/58f197ee-2586-441b-b591-66090ab5dd32-kube-api-access-m5bx5\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:17.007024 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:17.006978    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls podName:58f197ee-2586-441b-b591-66090ab5dd32 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:17.506960567 +0000 UTC m=+70.557889902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-z5mbc" (UID: "58f197ee-2586-441b-b591-66090ab5dd32") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:17.007536 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.007519    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/58f197ee-2586-441b-b591-66090ab5dd32-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:17.016026 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.016003    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5bx5\" (UniqueName: \"kubernetes.io/projected/58f197ee-2586-441b-b591-66090ab5dd32-kube-api-access-m5bx5\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:17.018872 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.018836    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:17.080236 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.080196    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:17.107700 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.107667    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.107833 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.107706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-stats-auth\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.107833 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.107724    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmm42\" (UniqueName: \"kubernetes.io/projected/17037412-22f7-435b-9b7c-54d02129d8c5-kube-api-access-cmm42\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.107833 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.107776    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-default-certificate\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.107833 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.107792    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.107986 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:17.107835    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:17.607816741 +0000 UTC m=+70.658746073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : configmap references non-existent config key: service-ca.crt
Jul 20 09:58:17.107986 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:17.107897    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:58:17.107986 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:17.107954    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:17.60793822 +0000 UTC m=+70.658867555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : secret "router-metrics-certs-default" not found
Jul 20 09:58:17.110240 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.110211    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-stats-auth\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.110376 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.110240    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-default-certificate\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.117614 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.117588    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmm42\" (UniqueName: \"kubernetes.io/projected/17037412-22f7-435b-9b7c-54d02129d8c5-kube-api-access-cmm42\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.511150 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.511064    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:17.511320 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:17.511211    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:17.511320 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:17.511303    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls podName:58f197ee-2586-441b-b591-66090ab5dd32 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:18.511284523 +0000 UTC m=+71.562213856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-z5mbc" (UID: "58f197ee-2586-441b-b591-66090ab5dd32") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:17.612185 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.612143    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.612361 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.612316    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:17.612361 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:17.612347    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:18.612324022 +0000 UTC m=+71.663253359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : configmap references non-existent config key: service-ca.crt
Jul 20 09:58:17.612447 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:17.612412    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:58:17.612484 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:17.612458    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:18.612445037 +0000 UTC m=+71.663374371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : secret "router-metrics-certs-default" not found
Jul 20 09:58:17.804629 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.804520    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5r7zg" event={"ID":"3796601d-5123-4213-8c2d-453d6212d472","Type":"ContainerStarted","Data":"cb92885263b6382f93d1fe7ea4f269255d58fb820f6340acf2454b4c1e6f003a"}
Jul 20 09:58:17.805909 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.805889    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q7zvj" event={"ID":"1d7b7476-9151-41f4-a037-54a40c87cdc4","Type":"ContainerStarted","Data":"07ee6a70f08273b62e789e004f2d34254d4e968944a8fa2e56b406f8eb823f31"}
Jul 20 09:58:17.806036 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.806020    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:58:17.828365 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.828310    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5r7zg" podStartSLOduration=65.119020982 podStartE2EDuration="1m9.828287066s" totalImagesPullingTime="4.709266084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:17.827900445 +0000 UTC m=+70.878829800" watchObservedRunningTime="2026-07-20 09:58:17.828287066 +0000 UTC m=+70.879216420"
Jul 20 09:58:17.852317 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:17.852266    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q7zvj" podStartSLOduration=66.164074157 podStartE2EDuration="1m10.852226093s" totalImagesPullingTime="4.688151936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:17.851851143 +0000 UTC m=+70.902780497" watchObservedRunningTime="2026-07-20 09:58:17.852226093 +0000 UTC m=+70.903155448"
Jul 20 09:58:18.518219 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:18.518170    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:18.518422 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:18.518356    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:18.518422 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:18.518423    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls podName:58f197ee-2586-441b-b591-66090ab5dd32 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:20.51840702 +0000 UTC m=+73.569336356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-z5mbc" (UID: "58f197ee-2586-441b-b591-66090ab5dd32") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:18.618674 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:18.618642    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:18.618847 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:18.618701    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:18.618847 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:18.618793    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:58:18.618847 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:18.618821    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:20.618808135 +0000 UTC m=+73.669737471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : configmap references non-existent config key: service-ca.crt
Jul 20 09:58:18.618847 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:18.618847    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:20.618833979 +0000 UTC m=+73.669763312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : secret "router-metrics-certs-default" not found
Jul 20 09:58:19.909035 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.908999    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8x59k"]
Jul 20 09:58:19.911752 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.911735    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:19.915516 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.915490    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 20 09:58:19.915516 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.915507    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 20 09:58:19.916809 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.916788    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nffcc\""
Jul 20 09:58:19.916950 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.916819    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 20 09:58:19.916950 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.916831    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 20 09:58:19.925461 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.925439    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8x59k"]
Jul 20 09:58:19.926737 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.926714    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/903c240d-9f9b-41d5-a837-95aa6feff967-crio-socket\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:19.926837 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.926741    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:19.926837 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.926765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:19.926837 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.926781    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g269k\" (UniqueName: \"kubernetes.io/projected/903c240d-9f9b-41d5-a837-95aa6feff967-kube-api-access-g269k\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:19.927004 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.926931    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/903c240d-9f9b-41d5-a837-95aa6feff967-data-volume\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:19.937362 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.937330    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:19.992404 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:19.992368    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:20.027506 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.027471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/903c240d-9f9b-41d5-a837-95aa6feff967-data-volume\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:20.027609 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.027566    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/903c240d-9f9b-41d5-a837-95aa6feff967-crio-socket\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:20.027609 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.027599    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:20.027683 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.027636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:20.027724 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.027694    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/903c240d-9f9b-41d5-a837-95aa6feff967-crio-socket\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:20.027773 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.027699    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:20.027773 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.027734    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:20.027773 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.027752    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g269k\" (UniqueName: \"kubernetes.io/projected/903c240d-9f9b-41d5-a837-95aa6feff967-kube-api-access-g269k\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:20.027773 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.027772    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:20.527755173 +0000 UTC m=+73.578684510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:20.027959 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.027856    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:20.527838022 +0000 UTC m=+73.578767358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:20.027959 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.027909    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/903c240d-9f9b-41d5-a837-95aa6feff967-data-volume\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:20.036590 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.036564    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g269k\" (UniqueName: \"kubernetes.io/projected/903c240d-9f9b-41d5-a837-95aa6feff967-kube-api-access-g269k\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:20.531480 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.531440    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:20.531480 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.531485    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:20.531757 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.531527    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:20.531757 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.531597    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:20.531757 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.531611    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:20.531757 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.531649    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:21.5316343 +0000 UTC m=+74.582563633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:20.531757 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.531676    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:21.531658318 +0000 UTC m=+74.582587652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:20.531757 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.531691    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:20.531757 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.531738    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls podName:58f197ee-2586-441b-b591-66090ab5dd32 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:24.531726744 +0000 UTC m=+77.582656077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-z5mbc" (UID: "58f197ee-2586-441b-b591-66090ab5dd32") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:20.632312 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.632269    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:20.632508 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:20.632349    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:20.632508 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.632413    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:58:20.632508 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.632475    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:24.632460228 +0000 UTC m=+77.683389561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : secret "router-metrics-certs-default" not found
Jul 20 09:58:20.632508 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:20.632490    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:24.632484125 +0000 UTC m=+77.683413458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : configmap references non-existent config key: service-ca.crt
Jul 20 09:58:21.538071 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:21.538011    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:21.538071 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:21.538072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:21.538567 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:21.538169    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:21.538567 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:21.538172    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:21.538567 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:21.538221    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:23.538206988 +0000 UTC m=+76.589136321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:21.538567 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:21.538235    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:23.538228673 +0000 UTC m=+76.589158006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:22.177897 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:22.177869    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tfh95_a0274aa1-d4b8-463c-8f61-d888fe04e631/dns-node-resolver/0.log"
Jul 20 09:58:22.577867 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:22.577785    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cm5ps_58dc4f4e-0a78-4b1d-af07-30685a4498d0/node-ca/0.log"
Jul 20 09:58:23.553490 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:23.553448    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:23.553490 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:23.553494    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:23.553706 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:23.553601    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:23.553706 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:23.553624    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:23.553706 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:23.553666    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:27.55365067 +0000 UTC m=+80.604580006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:23.553706 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:23.553680    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:27.553673898 +0000 UTC m=+80.604603230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:23.587306 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:23.587262    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jgts6_630a07a2-1674-43f8-82d6-08df700b040c/serve-healthcheck-canary/0.log"
Jul 20 09:58:24.563280 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:24.563215    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:24.563475 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:24.563373    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:24.563475 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:24.563444    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls podName:58f197ee-2586-441b-b591-66090ab5dd32 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:32.563428013 +0000 UTC m=+85.614357346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-z5mbc" (UID: "58f197ee-2586-441b-b591-66090ab5dd32") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:24.663913 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:24.663860    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:24.664293 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:24.663936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:24.664293 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:24.664056    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:58:24.664293 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:24.664069    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:32.664049181 +0000 UTC m=+85.714978517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : configmap references non-existent config key: service-ca.crt
Jul 20 09:58:24.664293 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:24.664114    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:32.664102221 +0000 UTC m=+85.715031561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : secret "router-metrics-certs-default" not found
Jul 20 09:58:27.587883 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:27.587839    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:27.588386 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:27.587944    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:27.588386 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:27.587979    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:27.588386 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:27.588039    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:27.588386 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:27.588064    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:35.588043894 +0000 UTC m=+88.638973246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:27.588386 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:27.588084    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:35.58807402 +0000 UTC m=+88.639003352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:29.099173 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:29.099138    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:29.152819 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:29.152732    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:31.135705 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:31.135672    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:31.242138 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:31.242104    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:31.323005 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:31.322968    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:32.014495 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:32.014459    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:32.072021 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:32.071984    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:32.431097 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:32.431066    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:32.485351 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:32.485318    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:32.626119 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:32.626075    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:32.626346 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:32.626217    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:32.626346 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:32.626312    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls podName:58f197ee-2586-441b-b591-66090ab5dd32 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:48.626290287 +0000 UTC m=+101.677219619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-z5mbc" (UID: "58f197ee-2586-441b-b591-66090ab5dd32") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:32.722765 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:32.722668    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:32.727464 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:32.727442    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:32.727565 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:32.727547    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:32.727648 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:32.727632    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:48.72760167 +0000 UTC m=+101.778531020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : configmap references non-existent config key: service-ca.crt
Jul 20 09:58:32.727699 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:32.727677    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:58:32.727744 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:32.727732    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs podName:17037412-22f7-435b-9b7c-54d02129d8c5 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:48.727718318 +0000 UTC m=+101.778647650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs") pod "router-default-67c67595d6-ltx5q" (UID: "17037412-22f7-435b-9b7c-54d02129d8c5") : secret "router-metrics-certs-default" not found
Jul 20 09:58:32.814537 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:32.814497    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:33.134019 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:33.133980    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:33.235364 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:33.235324    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:34.362014 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:34.361975    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:34.426012 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:34.425975    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:35.357906 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:35.357874    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:35.648113 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:35.648020    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:35.648113 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:35.648059    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:35.648668 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:35.648188    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:35.648668 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:58:35.648280    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls podName:903c240d-9f9b-41d5-a837-95aa6feff967 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:51.648241125 +0000 UTC m=+104.699170461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x59k" (UID: "903c240d-9f9b-41d5-a837-95aa6feff967") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:35.648766 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:35.648691    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/903c240d-9f9b-41d5-a837-95aa6feff967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:38.236258 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:38.236209    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:38.803022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:38.802987    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:38.907282 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:38.907230    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:39.007636 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:39.007591    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:39.127092 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:39.127053    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:39.326372 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:39.326338    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:39.438744 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:39.438662    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:39.533371 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:39.533339    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:40.283728 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:40.283684    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:40.870443 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:40.870408    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:41.074345 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:41.074310    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:41.477701 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:41.477657    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:41.552804 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:41.552768    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:41.675848 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:41.675809    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:41.872520 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:41.872491    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:42.484667 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:42.484618    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:42.680726 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:42.680685    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.064357 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:43.064313    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.272123 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:43.272092    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.474497 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:43.474444    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.538923 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:43.538887    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.677197 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:43.677156    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.884060 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:43.884026    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:44.890268 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:44.890210    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:45.074134 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:45.074105    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:45.492421 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:45.492369    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:45.681923 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:45.681890    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:46.277069 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:46.277029    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:46.356212 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:46.356164    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:46.523744 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:46.523709    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:46.585012 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:46.584898    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:46.682008 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:46.681972    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:46.738353 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:46.738311    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:58:46.740856 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:46.740831    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3-metrics-tls\") pod \"dns-default-mlj89\" (UID: \"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3\") " pod="openshift-dns/dns-default-mlj89"
Jul 20 09:58:47.035593 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:47.035560    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l5b2l\""
Jul 20 09:58:47.042557 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:47.042529    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mlj89"
Jul 20 09:58:47.089922 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:47.089885    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:47.192956 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:47.192911    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:47.199072 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:47.199044    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mlj89"]
Jul 20 09:58:47.202614 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:58:47.202584    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d0e67f_d74d_4e9b_8769_fdc9e2b56fa3.slice/crio-3d5665016383e586bfba826d1f1a0094184b52c6486338c7e109db38e1198cc9 WatchSource:0}: Error finding container 3d5665016383e586bfba826d1f1a0094184b52c6486338c7e109db38e1198cc9: Status 404 returned error can't find the container with id 3d5665016383e586bfba826d1f1a0094184b52c6486338c7e109db38e1198cc9
Jul 20 09:58:47.474029 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:47.473981    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:47.675540 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:47.675503    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:47.879406 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:47.879358    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mlj89" event={"ID":"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3","Type":"ContainerStarted","Data":"3d5665016383e586bfba826d1f1a0094184b52c6486338c7e109db38e1198cc9"}
Jul 20 09:58:48.267097 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.267008    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:48.473455 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.473415    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:48.656108 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.656059    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:48.658986 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.658925    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58f197ee-2586-441b-b591-66090ab5dd32-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-z5mbc\" (UID: \"58f197ee-2586-441b-b591-66090ab5dd32\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:48.757304 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.757232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:48.757463 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.757345    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:48.757948 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.757922    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17037412-22f7-435b-9b7c-54d02129d8c5-service-ca-bundle\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:48.759559 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.759536    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17037412-22f7-435b-9b7c-54d02129d8c5-metrics-certs\") pod \"router-default-67c67595d6-ltx5q\" (UID: \"17037412-22f7-435b-9b7c-54d02129d8c5\") " pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:48.811154 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.811119    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q7zvj"
Jul 20 09:58:48.847288 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.847231    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"
Jul 20 09:58:48.882315 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.882279    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:48.968757 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.968728    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:48.984409 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:48.984382    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc"]
Jul 20 09:58:48.988013 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:58:48.987984    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f197ee_2586_441b_b591_66090ab5dd32.slice/crio-49959bb7b8379704312c8fefe88c0ddaee0018170d8beb8e321178167a9fbe1d WatchSource:0}: Error finding container 49959bb7b8379704312c8fefe88c0ddaee0018170d8beb8e321178167a9fbe1d: Status 404 returned error can't find the container with id 49959bb7b8379704312c8fefe88c0ddaee0018170d8beb8e321178167a9fbe1d
Jul 20 09:58:49.111227 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.111187    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67c67595d6-ltx5q"]
Jul 20 09:58:49.180438 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:58:49.180364    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17037412_22f7_435b_9b7c_54d02129d8c5.slice/crio-60dccd3fc4db3994f4a2233c26925825e81ed30d57dd78ed374c14095829b0a6 WatchSource:0}: Error finding container 60dccd3fc4db3994f4a2233c26925825e81ed30d57dd78ed374c14095829b0a6: Status 404 returned error can't find the container with id 60dccd3fc4db3994f4a2233c26925825e81ed30d57dd78ed374c14095829b0a6
Jul 20 09:58:49.268768 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.268725    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:49.468614 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.468581    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:49.887780 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.887741    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67c67595d6-ltx5q" event={"ID":"17037412-22f7-435b-9b7c-54d02129d8c5","Type":"ContainerStarted","Data":"2f933fc011a3e809a620cf75cb5c8c2a40f13441c6d3f1f9a25d9a918904ebfc"}
Jul 20 09:58:49.888293 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.887804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67c67595d6-ltx5q" event={"ID":"17037412-22f7-435b-9b7c-54d02129d8c5","Type":"ContainerStarted","Data":"60dccd3fc4db3994f4a2233c26925825e81ed30d57dd78ed374c14095829b0a6"}
Jul 20 09:58:49.889812 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.889778    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mlj89" event={"ID":"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3","Type":"ContainerStarted","Data":"8b32af43aaff908603fc121f861541b7797d640187600bbf6caffae50730c015"}
Jul 20 09:58:49.889949 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.889836    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mlj89"
Jul 20 09:58:49.889949 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.889861    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mlj89" event={"ID":"c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3","Type":"ContainerStarted","Data":"dfd3999266a216fbb7d62ef918eb810c0d826291ce41d46b9430f5f91ff206d0"}
Jul 20 09:58:49.890992 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.890968    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc" event={"ID":"58f197ee-2586-441b-b591-66090ab5dd32","Type":"ContainerStarted","Data":"49959bb7b8379704312c8fefe88c0ddaee0018170d8beb8e321178167a9fbe1d"}
Jul 20 09:58:49.934082 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.934010    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-67c67595d6-ltx5q" podStartSLOduration=33.933989205 podStartE2EDuration="33.933989205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:49.912526127 +0000 UTC m=+102.963455483" watchObservedRunningTime="2026-07-20 09:58:49.933989205 +0000 UTC m=+102.984918565"
Jul 20 09:58:49.934763 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.934713    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mlj89" podStartSLOduration=65.923189151 podStartE2EDuration="1m7.934699607s" totalImagesPullingTime="2.011510456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:49.933570824 +0000 UTC m=+102.984500180" watchObservedRunningTime="2026-07-20 09:58:49.934699607 +0000 UTC m=+102.985628940"
Jul 20 09:58:49.969137 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.969103    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:49.971967 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:49.971943    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:50.894974 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:50.894944    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:50.896267 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:50.896232    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-67c67595d6-ltx5q"
Jul 20 09:58:51.436313 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.436212    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:51.536265 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.536208    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:51.636062 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.636027    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:51.681297 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.681229    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:51.683652 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.683608    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/903c240d-9f9b-41d5-a837-95aa6feff967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x59k\" (UID: \"903c240d-9f9b-41d5-a837-95aa6feff967\") " pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:51.689664 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.689589    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:51.720670 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.720638    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8x59k"
Jul 20 09:58:51.877283 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.877227    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8x59k"]
Jul 20 09:58:51.880585 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:58:51.880547    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903c240d_9f9b_41d5_a837_95aa6feff967.slice/crio-42e1dab5dfb59ce1cacac9555c09f83134299fbad16652f5bbb3008e8780b6fa WatchSource:0}: Error finding container 42e1dab5dfb59ce1cacac9555c09f83134299fbad16652f5bbb3008e8780b6fa: Status 404 returned error can't find the container with id 42e1dab5dfb59ce1cacac9555c09f83134299fbad16652f5bbb3008e8780b6fa
Jul 20 09:58:51.899737 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.899700    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc" event={"ID":"58f197ee-2586-441b-b591-66090ab5dd32","Type":"ContainerStarted","Data":"3dae794d26f53c304059e20aa18d03dc3527023725d2cad03313e8ca9e638578"}
Jul 20 09:58:51.900828 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.900804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x59k" event={"ID":"903c240d-9f9b-41d5-a837-95aa6feff967","Type":"ContainerStarted","Data":"42e1dab5dfb59ce1cacac9555c09f83134299fbad16652f5bbb3008e8780b6fa"}
Jul 20 09:58:51.917785 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:51.917736    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-z5mbc" podStartSLOduration=33.810976656 podStartE2EDuration="35.917716954s" totalImagesPullingTime="2.106740298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:51.91734687 +0000 UTC m=+104.968276225" watchObservedRunningTime="2026-07-20 09:58:51.917716954 +0000 UTC m=+104.968646309"
Jul 20 09:58:53.201052 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:53.201008    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:53.300922 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:53.300865    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:53.506863 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:53.506773    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:53.878833 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:53.878798    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:53.937672 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:53.937627    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:54.275807 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:54.275714    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:54.333048 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:54.333002    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.088143 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.088096    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.293536 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.293491    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.696737 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.696693    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.847927 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.847890    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"]
Jul 20 09:58:55.897433 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.897403    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"]
Jul 20 09:58:55.897590 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.897545    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:55.900369 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.900310    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 20 09:58:55.902057 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.902033    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 20 09:58:55.902199 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.902083    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 20 09:58:55.902288 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.902224    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 20 09:58:55.942968 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.942903    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-6pxc9"]
Jul 20 09:58:55.949538 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.949452    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-687f897475-jbl2v"]
Jul 20 09:58:55.949690 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.949594    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-6pxc9"
Jul 20 09:58:55.952500 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.952475    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 20 09:58:55.952975 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.952956    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 20 09:58:55.952975 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.952966    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4bmw7\""
Jul 20 09:58:55.953964 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.953945    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:55.957054 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.957029    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 20 09:58:55.957054 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.957029    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 20 09:58:55.957219 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.957139    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 20 09:58:55.961359 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.961326    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bp7zt\""
Jul 20 09:58:55.962520 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.962431    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-6pxc9"]
Jul 20 09:58:55.965134 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.965113    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 20 09:58:55.975988 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.975950    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.983950 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:55.983915    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687f897475-jbl2v"]
Jul 20 09:58:56.020756 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.020711    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-installation-pull-secrets\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.020756 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.020762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-trusted-ca\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.020971 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.020814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbmb\" (UniqueName: \"kubernetes.io/projected/00319ed1-836b-4b3a-a2e8-da8d38bb3a66-kube-api-access-njbmb\") pod \"downloads-5bc67dcd46-6pxc9\" (UID: \"00319ed1-836b-4b3a-a2e8-da8d38bb3a66\") " pod="openshift-console/downloads-5bc67dcd46-6pxc9"
Jul 20 09:58:56.020971 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.020859    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-registry-tls\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.020971 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.020915    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-ca-trust-extracted\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.020971 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.020940    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdb7l\" (UniqueName: \"kubernetes.io/projected/131fe800-1080-4b5d-b7f8-d93c8df3e113-kube-api-access-fdb7l\") pod \"klusterlet-addon-workmgr-6684df946f-9rr4q\" (UID: \"131fe800-1080-4b5d-b7f8-d93c8df3e113\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:56.020971 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.020960    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-image-registry-private-configuration\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.021156 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.021009    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-registry-certificates\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.021156 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.021033    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/131fe800-1080-4b5d-b7f8-d93c8df3e113-klusterlet-config\") pod \"klusterlet-addon-workmgr-6684df946f-9rr4q\" (UID: \"131fe800-1080-4b5d-b7f8-d93c8df3e113\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:56.021156 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.021074    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/131fe800-1080-4b5d-b7f8-d93c8df3e113-tmp\") pod \"klusterlet-addon-workmgr-6684df946f-9rr4q\" (UID: \"131fe800-1080-4b5d-b7f8-d93c8df3e113\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:56.021156 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.021093    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-bound-sa-token\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.021156 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.021120    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdls\" (UniqueName: \"kubernetes.io/projected/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-kube-api-access-zmdls\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.059023 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.058982    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:56.122468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.122429    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-registry-certificates\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.122468 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.122468    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/131fe800-1080-4b5d-b7f8-d93c8df3e113-klusterlet-config\") pod \"klusterlet-addon-workmgr-6684df946f-9rr4q\" (UID: \"131fe800-1080-4b5d-b7f8-d93c8df3e113\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:56.122690 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.122660    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/131fe800-1080-4b5d-b7f8-d93c8df3e113-tmp\") pod \"klusterlet-addon-workmgr-6684df946f-9rr4q\" (UID: \"131fe800-1080-4b5d-b7f8-d93c8df3e113\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:56.122747 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.122698    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-bound-sa-token\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.122747 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.122727    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdls\" (UniqueName: \"kubernetes.io/projected/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-kube-api-access-zmdls\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.122843 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.122765    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-installation-pull-secrets\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.123226 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.122963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-trusted-ca\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.123226 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.123015    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbmb\" (UniqueName: \"kubernetes.io/projected/00319ed1-836b-4b3a-a2e8-da8d38bb3a66-kube-api-access-njbmb\") pod \"downloads-5bc67dcd46-6pxc9\" (UID: \"00319ed1-836b-4b3a-a2e8-da8d38bb3a66\") " pod="openshift-console/downloads-5bc67dcd46-6pxc9"
Jul 20 09:58:56.123226 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.123055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-registry-tls\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.123226 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.123057    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/131fe800-1080-4b5d-b7f8-d93c8df3e113-tmp\") pod \"klusterlet-addon-workmgr-6684df946f-9rr4q\" (UID: \"131fe800-1080-4b5d-b7f8-d93c8df3e113\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:56.123226 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.123124    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-ca-trust-extracted\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.123226 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.123164    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdb7l\" (UniqueName: \"kubernetes.io/projected/131fe800-1080-4b5d-b7f8-d93c8df3e113-kube-api-access-fdb7l\") pod \"klusterlet-addon-workmgr-6684df946f-9rr4q\" (UID: \"131fe800-1080-4b5d-b7f8-d93c8df3e113\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:56.123226 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.123192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-image-registry-private-configuration\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.123687 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.123319    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-registry-certificates\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.124004 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.123944    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-trusted-ca\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.124102 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.124007    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-ca-trust-extracted\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.125442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.125413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-installation-pull-secrets\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.125589 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.125568    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/131fe800-1080-4b5d-b7f8-d93c8df3e113-klusterlet-config\") pod \"klusterlet-addon-workmgr-6684df946f-9rr4q\" (UID: \"131fe800-1080-4b5d-b7f8-d93c8df3e113\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:56.125747 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.125732    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-registry-tls\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.125845 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.125822    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-image-registry-private-configuration\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.138504 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.138480    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdb7l\" (UniqueName: \"kubernetes.io/projected/131fe800-1080-4b5d-b7f8-d93c8df3e113-kube-api-access-fdb7l\") pod \"klusterlet-addon-workmgr-6684df946f-9rr4q\" (UID: \"131fe800-1080-4b5d-b7f8-d93c8df3e113\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:56.138760 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.138736    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbmb\" (UniqueName: \"kubernetes.io/projected/00319ed1-836b-4b3a-a2e8-da8d38bb3a66-kube-api-access-njbmb\") pod \"downloads-5bc67dcd46-6pxc9\" (UID: \"00319ed1-836b-4b3a-a2e8-da8d38bb3a66\") " pod="openshift-console/downloads-5bc67dcd46-6pxc9"
Jul 20 09:58:56.138841 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.138814    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-bound-sa-token\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.138991 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.138973    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdls\" (UniqueName: \"kubernetes.io/projected/e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2-kube-api-access-zmdls\") pod \"image-registry-687f897475-jbl2v\" (UID: \"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2\") " pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.208893 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.208812    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:58:56.263107 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.263068    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-6pxc9"
Jul 20 09:58:56.269137 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.269101    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:56.922500 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.922468    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"]
Jul 20 09:58:56.923660 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:56.923567    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x59k" event={"ID":"903c240d-9f9b-41d5-a837-95aa6feff967","Type":"ContainerStarted","Data":"413d1c84b491894aca4b81164f45aea64f532adcadd906089a91f5d190edccb0"}
Jul 20 09:58:56.927459 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:58:56.927427    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131fe800_1080_4b5d_b7f8_d93c8df3e113.slice/crio-31b5af8cdc3d2c1ec8ec0a138ceb1d3dbd6080e0526a0c850b58557978ccf2d1 WatchSource:0}: Error finding container 31b5af8cdc3d2c1ec8ec0a138ceb1d3dbd6080e0526a0c850b58557978ccf2d1: Status 404 returned error can't find the container with id 31b5af8cdc3d2c1ec8ec0a138ceb1d3dbd6080e0526a0c850b58557978ccf2d1
Jul 20 09:58:57.152536 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:57.152504    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-6pxc9"]
Jul 20 09:58:57.159564 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:57.157005    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687f897475-jbl2v"]
Jul 20 09:58:57.933890 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:57.933841    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-6pxc9" event={"ID":"00319ed1-836b-4b3a-a2e8-da8d38bb3a66","Type":"ContainerStarted","Data":"f9f6cd427333a91f6cd773ead09aed744350872f0b5de316ef7f970e38a3a64a"}
Jul 20 09:58:57.935502 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:57.935469    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q" event={"ID":"131fe800-1080-4b5d-b7f8-d93c8df3e113","Type":"ContainerStarted","Data":"31b5af8cdc3d2c1ec8ec0a138ceb1d3dbd6080e0526a0c850b58557978ccf2d1"}
Jul 20 09:58:57.938051 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:57.938023    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687f897475-jbl2v" event={"ID":"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2","Type":"ContainerStarted","Data":"9a39581eaf1db891a8cbbcd655226e2385f98983d36148afe3c4b60bf5ee5476"}
Jul 20 09:58:57.938181 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:57.938070    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687f897475-jbl2v" event={"ID":"e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2","Type":"ContainerStarted","Data":"5fa01049f8a2f7031019e1c29356f7fbe737280b777b474b86a912e4c0dead3e"}
Jul 20 09:58:57.938261 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:57.938185    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:58:57.969585 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:57.969023    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-687f897475-jbl2v" podStartSLOduration=2.968999427 podStartE2EDuration="2.968999427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:57.966440839 +0000 UTC m=+111.017370206" watchObservedRunningTime="2026-07-20 09:58:57.968999427 +0000 UTC m=+111.019928766"
Jul 20 09:58:59.337768 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.336630    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cc85c975b-pnwkl"]
Jul 20 09:58:59.340329 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.340274    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:59.343002 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.342322    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.347097 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.346917    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 20 09:58:59.347658 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.347624    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tcc2t\""
Jul 20 09:58:59.347968 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.347936    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 20 09:58:59.348075 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.347974    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 20 09:58:59.348258 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.348215    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 20 09:58:59.348258 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.348235    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 20 09:58:59.353438 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.353360    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cc85c975b-pnwkl"]
Jul 20 09:58:59.407572 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.407463    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:59.454451 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.454117    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-console-config\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.454451 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.454183    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-service-ca\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.454451 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.454289    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-serving-cert\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.454451 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.454323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-oauth-serving-cert\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.454451 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.454369    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-oauth-config\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.454451 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.454392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pn5c\" (UniqueName: \"kubernetes.io/projected/14d96d5e-e056-42cc-814f-0986b3a0e015-kube-api-access-2pn5c\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.555116 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.555064    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-oauth-config\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.555116 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.555120    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pn5c\" (UniqueName: \"kubernetes.io/projected/14d96d5e-e056-42cc-814f-0986b3a0e015-kube-api-access-2pn5c\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.555491 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.555154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-console-config\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.555491 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.555188    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-service-ca\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.555491 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.555274    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-serving-cert\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.555671 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.555641    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-oauth-serving-cert\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.556482 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.556432    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-oauth-serving-cert\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.556619 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.556589    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-console-config\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.556679 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.556656    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-service-ca\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.559263 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.559197    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-oauth-config\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.560024 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.559996    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-serving-cert\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.570415 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.570372    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pn5c\" (UniqueName: \"kubernetes.io/projected/14d96d5e-e056-42cc-814f-0986b3a0e015-kube-api-access-2pn5c\") pod \"console-5cc85c975b-pnwkl\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") " pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.656497 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.656213    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:58:59.897719 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:58:59.897435    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mlj89"
Jul 20 09:59:01.511331 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:01.511292    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:01.568333 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:01.568294    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:03.538677 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.538638    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:03.543840 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.543812    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84f57cd5c8-nxt7q"]
Jul 20 09:59:03.547095 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.547075    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.558477 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.558445    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cc85c975b-pnwkl"]
Jul 20 09:59:03.559422 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.559176    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 20 09:59:03.560742 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:59:03.560712    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d96d5e_e056_42cc_814f_0986b3a0e015.slice/crio-5af2a876787837ed98b82471f2ed6cc63e0d4da5fa75477f24ebc83cf704a65f WatchSource:0}: Error finding container 5af2a876787837ed98b82471f2ed6cc63e0d4da5fa75477f24ebc83cf704a65f: Status 404 returned error can't find the container with id 5af2a876787837ed98b82471f2ed6cc63e0d4da5fa75477f24ebc83cf704a65f
Jul 20 09:59:03.570722 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.570624    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f57cd5c8-nxt7q"]
Jul 20 09:59:03.613029 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.612986    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:03.694092 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.694008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-service-ca\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.694092 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.694068    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-oauth-serving-cert\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.694375 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.694167    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-trusted-ca-bundle\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.694375 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.694218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-oauth-config\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.694375 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.694292    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75znx\" (UniqueName: \"kubernetes.io/projected/6bbe1805-14b5-4c69-94be-3e728eaea4af-kube-api-access-75znx\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.694375 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.694334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-serving-cert\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.694375 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.694362    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-config\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.795126 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.795093    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-serving-cert\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.795126 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.795134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-config\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.795400 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.795168    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-service-ca\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.795400 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.795209    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-oauth-serving-cert\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.795400 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.795276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-trusted-ca-bundle\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.795400 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.795373    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-oauth-config\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.795619 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.795422    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75znx\" (UniqueName: \"kubernetes.io/projected/6bbe1805-14b5-4c69-94be-3e728eaea4af-kube-api-access-75znx\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.796432 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.796407    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-service-ca\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.796551 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.796510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-config\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.796551 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.796522    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-oauth-serving-cert\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.796551 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.796537    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-trusted-ca-bundle\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.798558 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.798529    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-oauth-config\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.798695 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.798539    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-serving-cert\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.808923 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.808888    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75znx\" (UniqueName: \"kubernetes.io/projected/6bbe1805-14b5-4c69-94be-3e728eaea4af-kube-api-access-75znx\") pod \"console-84f57cd5c8-nxt7q\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") " pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.861073 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.861037    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:03.959745 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.959683    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:03.965459 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.965337    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q" event={"ID":"131fe800-1080-4b5d-b7f8-d93c8df3e113","Type":"ContainerStarted","Data":"546ea4fbb4d940a847e9ed02cb6dce7e790e417dc6fb809e05008bc1ee061912"}
Jul 20 09:59:03.965459 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.965415    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:59:03.967670 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.967345    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc85c975b-pnwkl" event={"ID":"14d96d5e-e056-42cc-814f-0986b3a0e015","Type":"ContainerStarted","Data":"5af2a876787837ed98b82471f2ed6cc63e0d4da5fa75477f24ebc83cf704a65f"}
Jul 20 09:59:03.967670 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.967615    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q"
Jul 20 09:59:03.970281 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.970232    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x59k" event={"ID":"903c240d-9f9b-41d5-a837-95aa6feff967","Type":"ContainerStarted","Data":"84d6c7f134e30b88a8071d486cef8ce85037e52501e042e982790f3776d6bf58"}
Jul 20 09:59:03.987078 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:03.987012    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684df946f-9rr4q" podStartSLOduration=2.489554885 podStartE2EDuration="8.986993986s" totalImagesPullingTime="6.497439101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:03.986278129 +0000 UTC m=+117.037207479" watchObservedRunningTime="2026-07-20 09:59:03.986993986 +0000 UTC m=+117.037923333"
Jul 20 09:59:04.023357 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:04.023324    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f57cd5c8-nxt7q"]
Jul 20 09:59:04.026860 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:59:04.026823    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbe1805_14b5_4c69_94be_3e728eaea4af.slice/crio-3409fb391cc0659c7ee038a5bd61fce41910597fca33509946cd0e854e120005 WatchSource:0}: Error finding container 3409fb391cc0659c7ee038a5bd61fce41910597fca33509946cd0e854e120005: Status 404 returned error can't find the container with id 3409fb391cc0659c7ee038a5bd61fce41910597fca33509946cd0e854e120005
Jul 20 09:59:04.037032 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:04.036944    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:04.256290 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:04.256187    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:04.315796 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:04.315704    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:04.657857 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:04.657801    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:04.739569 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:04.739518    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:04.978537 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:04.978284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f57cd5c8-nxt7q" event={"ID":"6bbe1805-14b5-4c69-94be-3e728eaea4af","Type":"ContainerStarted","Data":"3409fb391cc0659c7ee038a5bd61fce41910597fca33509946cd0e854e120005"}
Jul 20 09:59:04.978537 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:04.978396    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:07.249006 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:07.248907    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:07.304839 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:07.304798    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:07.886955 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:07.886904    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:07.947115 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:07.947035    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:08.067850 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:08.067812    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:09.036428 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:09.036380    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:09.137632 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:09.137581    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:10.823995 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:10.823956    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:10.948082 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:10.948038    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.048992 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.048954    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.127743 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.127709    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"]
Jul 20 09:59:11.131400 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.131376    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.134761 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.134735    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 20 09:59:11.136199 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.136174    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-57wwq\""
Jul 20 09:59:11.136429 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.136413    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 20 09:59:11.136493 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.136431    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 20 09:59:11.143367 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.143339    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"]
Jul 20 09:59:11.164949 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.164901    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.165173 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.164964    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsftq\" (UniqueName: \"kubernetes.io/projected/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-kube-api-access-qsftq\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.165173 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.165074    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.165173 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.165102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.184744 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.184703    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.194238 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.194208    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-njr9r"]
Jul 20 09:59:11.197617 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.197595    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.200527 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.200492    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 20 09:59:11.201031 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.200998    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-b8ppt\""
Jul 20 09:59:11.201153 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.201053    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 20 09:59:11.201218 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.201158    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 20 09:59:11.216975 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.216941    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-njr9r"]
Jul 20 09:59:11.245868 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.245832    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lltk8"]
Jul 20 09:59:11.249316 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.249284    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.254207 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.254155    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 20 09:59:11.254382 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.254213    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 20 09:59:11.254565 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.254546    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 20 09:59:11.254866 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.254847    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2n548\""
Jul 20 09:59:11.265979 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.265944    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-textfile\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.266147 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.265992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/743df18d-8914-47c2-b93d-2245a9bd400f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.266147 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266024    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-accelerators-collector-config\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.266147 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.266147 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/743df18d-8914-47c2-b93d-2245a9bd400f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.266442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266164    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.266442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266188    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/743df18d-8914-47c2-b93d-2245a9bd400f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.266442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266220    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/743df18d-8914-47c2-b93d-2245a9bd400f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.266442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266271    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2c315b98-9976-4328-bad4-d33913f17621-root\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.266442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-wtmp\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.266442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266321    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-tls\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.266442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266359    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ctv5\" (UniqueName: \"kubernetes.io/projected/2c315b98-9976-4328-bad4-d33913f17621-kube-api-access-4ctv5\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.266442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266404    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/743df18d-8914-47c2-b93d-2245a9bd400f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.266442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c315b98-9976-4328-bad4-d33913f17621-sys\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.266871 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266453    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.266871 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266493    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.266871 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266518    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c315b98-9976-4328-bad4-d33913f17621-metrics-client-ca\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.266871 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266572    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzvn\" (UniqueName: \"kubernetes.io/projected/743df18d-8914-47c2-b93d-2245a9bd400f-kube-api-access-jfzvn\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.266871 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.266640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsftq\" (UniqueName: \"kubernetes.io/projected/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-kube-api-access-qsftq\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.267290 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.267263    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.269458 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.269411    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.269571 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.269484    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.273451 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.273411    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.287056 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.287025    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsftq\" (UniqueName: \"kubernetes.io/projected/1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1-kube-api-access-qsftq\") pod \"openshift-state-metrics-79f6958cdf-kqrjp\" (UID: \"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.368059 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368018    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-textfile\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368288 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368067    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/743df18d-8914-47c2-b93d-2245a9bd400f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.368288 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-accelerators-collector-config\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368432 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368303    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/743df18d-8914-47c2-b93d-2245a9bd400f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.368432 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368360    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/743df18d-8914-47c2-b93d-2245a9bd400f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.368432 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368393    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/743df18d-8914-47c2-b93d-2245a9bd400f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.368432 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368420    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2c315b98-9976-4328-bad4-d33913f17621-root\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368627 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368444    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-wtmp\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368627 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-tls\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368627 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-textfile\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368627 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368509    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ctv5\" (UniqueName: \"kubernetes.io/projected/2c315b98-9976-4328-bad4-d33913f17621-kube-api-access-4ctv5\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368627 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368555    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/743df18d-8914-47c2-b93d-2245a9bd400f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.368627 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368563    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2c315b98-9976-4328-bad4-d33913f17621-root\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368627 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368577    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c315b98-9976-4328-bad4-d33913f17621-sys\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368977 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368635    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c315b98-9976-4328-bad4-d33913f17621-sys\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368977 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368719    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/743df18d-8914-47c2-b93d-2245a9bd400f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.368977 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368758    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-wtmp\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368977 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368859    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-accelerators-collector-config\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.368977 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/743df18d-8914-47c2-b93d-2245a9bd400f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.368977 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.368971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.369293 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.369024    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c315b98-9976-4328-bad4-d33913f17621-metrics-client-ca\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.369293 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.369069    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzvn\" (UniqueName: \"kubernetes.io/projected/743df18d-8914-47c2-b93d-2245a9bd400f-kube-api-access-jfzvn\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.369293 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.369269    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/743df18d-8914-47c2-b93d-2245a9bd400f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.369676 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.369601    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c315b98-9976-4328-bad4-d33913f17621-metrics-client-ca\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.370644 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.370584    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.371680 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.371626    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-tls\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.371984 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.371962    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/743df18d-8914-47c2-b93d-2245a9bd400f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.372505 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.372469    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c315b98-9976-4328-bad4-d33913f17621-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.372883 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.372837    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/743df18d-8914-47c2-b93d-2245a9bd400f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.378553 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.378475    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ctv5\" (UniqueName: \"kubernetes.io/projected/2c315b98-9976-4328-bad4-d33913f17621-kube-api-access-4ctv5\") pod \"node-exporter-lltk8\" (UID: \"2c315b98-9976-4328-bad4-d33913f17621\") " pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.378823 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.378790    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzvn\" (UniqueName: \"kubernetes.io/projected/743df18d-8914-47c2-b93d-2245a9bd400f-kube-api-access-jfzvn\") pod \"kube-state-metrics-56c5594674-njr9r\" (UID: \"743df18d-8914-47c2-b93d-2245a9bd400f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.443162 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.443038    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"
Jul 20 09:59:11.509516 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.509481    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r"
Jul 20 09:59:11.561576 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.561542    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lltk8"
Jul 20 09:59:11.570703 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.570595    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.879808 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:11.879769    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:12.145815 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:12.145730    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:12.247237 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:12.247183    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:12.446866 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:12.446689    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:12.718177 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:12.718092    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:12.825347 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:12.825297    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:14.045143 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.045096    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:14.155362 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.155323    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:14.168113 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.168079    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"]
Jul 20 09:59:14.241529 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.241489    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:14.276309 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.276276    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"]
Jul 20 09:59:14.276482 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.276406    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.280191 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.280161    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 20 09:59:14.280379 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.280161    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 20 09:59:14.280379 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.280310    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-co98nb2pd1i2u\""
Jul 20 09:59:14.280379 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.280319    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 20 09:59:14.280654 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.280640    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 20 09:59:14.280740 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.280719    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 20 09:59:14.281409 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.281392    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xsrsj\""
Jul 20 09:59:14.298872 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.298797    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-metrics-client-ca\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.298872 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.298849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-grpc-tls\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.299093 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.298925    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-tls\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.299093 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.298959    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4frd\" (UniqueName: \"kubernetes.io/projected/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-kube-api-access-p4frd\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.299093 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.298996    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.299093 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.299067    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.299284 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.299131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.299284 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.299160    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.351143 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.351100    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:14.400493 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.400438    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-metrics-client-ca\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.400493 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.400499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-grpc-tls\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.400752 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.400535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-tls\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.400752 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.400563    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4frd\" (UniqueName: \"kubernetes.io/projected/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-kube-api-access-p4frd\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.400752 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.400607    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.400752 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.400638    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.400752 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.400693    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.400752 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.400719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.401383 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.401351    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-metrics-client-ca\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.404465 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.403854    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.404465 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.403866    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.404465 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.404278    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.404465 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.404425    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.404952 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.404933    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-grpc-tls\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.405161 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.405142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-secret-thanos-querier-tls\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.411422 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.411398    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4frd\" (UniqueName: \"kubernetes.io/projected/e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd-kube-api-access-p4frd\") pod \"thanos-querier-5dd6b67cdf-5wcd5\" (UID: \"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd\") " pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:14.588596 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:14.588498    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:15.846319 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:15.846276    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:15.851759 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:15.851726    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"]
Jul 20 09:59:15.901695 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:15.901645    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cc85c975b-pnwkl"]
Jul 20 09:59:15.901695 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:15.901702    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"]
Jul 20 09:59:15.901914 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:15.901781    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:15.904494 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:15.904468    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 20 09:59:15.904770 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:15.904746    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-sc9w6\""
Jul 20 09:59:15.921122 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:15.921086    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:16.017495 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.017456    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbkjb\" (UniqueName: \"kubernetes.io/projected/080a463c-1129-46fd-99ac-e2c1505234a9-kube-api-access-nbkjb\") pod \"monitoring-plugin-7b7b7bc8bf-m7fmr\" (UID: \"080a463c-1129-46fd-99ac-e2c1505234a9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:16.017679 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.017621    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/080a463c-1129-46fd-99ac-e2c1505234a9-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-m7fmr\" (UID: \"080a463c-1129-46fd-99ac-e2c1505234a9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:16.119147 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.119103    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/080a463c-1129-46fd-99ac-e2c1505234a9-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-m7fmr\" (UID: \"080a463c-1129-46fd-99ac-e2c1505234a9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:16.119378 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.119182    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbkjb\" (UniqueName: \"kubernetes.io/projected/080a463c-1129-46fd-99ac-e2c1505234a9-kube-api-access-nbkjb\") pod \"monitoring-plugin-7b7b7bc8bf-m7fmr\" (UID: \"080a463c-1129-46fd-99ac-e2c1505234a9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:16.119378 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:59:16.119299    2555 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 20 09:59:16.119500 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:59:16.119386    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/080a463c-1129-46fd-99ac-e2c1505234a9-monitoring-plugin-cert podName:080a463c-1129-46fd-99ac-e2c1505234a9 nodeName:}" failed. No retries permitted until 2026-07-20 09:59:16.619363613 +0000 UTC m=+129.670292954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/080a463c-1129-46fd-99ac-e2c1505234a9-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-m7fmr" (UID: "080a463c-1129-46fd-99ac-e2c1505234a9") : secret "monitoring-plugin-cert" not found
Jul 20 09:59:16.129486 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.129454    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbkjb\" (UniqueName: \"kubernetes.io/projected/080a463c-1129-46fd-99ac-e2c1505234a9-kube-api-access-nbkjb\") pod \"monitoring-plugin-7b7b7bc8bf-m7fmr\" (UID: \"080a463c-1129-46fd-99ac-e2c1505234a9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:16.320720 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.320682    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:59:16.323579 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.323544    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed36d59b-3fc9-43ea-9506-f28d6955b7a3-metrics-certs\") pod \"network-metrics-daemon-p24w6\" (UID: \"ed36d59b-3fc9-43ea-9506-f28d6955b7a3\") " pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:59:16.359299 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.359259    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:16.607204 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.607180    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8dwl2\""
Jul 20 09:59:16.614646 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.614622    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p24w6"
Jul 20 09:59:16.623171 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.623145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/080a463c-1129-46fd-99ac-e2c1505234a9-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-m7fmr\" (UID: \"080a463c-1129-46fd-99ac-e2c1505234a9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:16.625547 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.625525    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/080a463c-1129-46fd-99ac-e2c1505234a9-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-m7fmr\" (UID: \"080a463c-1129-46fd-99ac-e2c1505234a9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:16.813371 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.813335    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:16.979264 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:16.978832    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:17.076110 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.069970    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f57cd5c8-nxt7q" event={"ID":"6bbe1805-14b5-4c69-94be-3e728eaea4af","Type":"ContainerStarted","Data":"4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e"}
Jul 20 09:59:17.077652 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.077569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lltk8" event={"ID":"2c315b98-9976-4328-bad4-d33913f17621","Type":"ContainerStarted","Data":"b387720e562baf6cb234fa2d2e1a501e45b1b7d0f8d2f99519c81bd0fde96dd7"}
Jul 20 09:59:17.081927 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.081832    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:17.136811 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.136324    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84f57cd5c8-nxt7q" podStartSLOduration=1.297672584 podStartE2EDuration="14.136301291s" totalImagesPullingTime="12.838628707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:17.136194696 +0000 UTC m=+130.187124052" watchObservedRunningTime="2026-07-20 09:59:17.136301291 +0000 UTC m=+130.187230647"
Jul 20 09:59:17.156003 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.152680    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp"]
Jul 20 09:59:17.156003 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.155106    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"]
Jul 20 09:59:17.157670 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:59:17.157638    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8eae26e_a89d_4b02_8ca0_cb4fc1ac19bd.slice/crio-c9cb7f049067c5d171dfc36ee92accd6e85939c5a8432913a14b707076f00860 WatchSource:0}: Error finding container c9cb7f049067c5d171dfc36ee92accd6e85939c5a8432913a14b707076f00860: Status 404 returned error can't find the container with id c9cb7f049067c5d171dfc36ee92accd6e85939c5a8432913a14b707076f00860
Jul 20 09:59:17.164263 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.164089    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-njr9r"]
Jul 20 09:59:17.168398 ip-10-0-129-35 kubenswrapper[2555]: W0720 09:59:17.168369    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743df18d_8914_47c2_b93d_2245a9bd400f.slice/crio-cd7f1afe05d6b8a29e103c63b897a849e06587415bca50fd02291faeeb0e3b57 WatchSource:0}: Error finding container cd7f1afe05d6b8a29e103c63b897a849e06587415bca50fd02291faeeb0e3b57: Status 404 returned error can't find the container with id cd7f1afe05d6b8a29e103c63b897a849e06587415bca50fd02291faeeb0e3b57
Jul 20 09:59:17.174662 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.174607    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8x59k" podStartSLOduration=33.352557481 podStartE2EDuration="58.174583903s" totalImagesPullingTime="24.822026422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:58:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:17.17413191 +0000 UTC m=+130.225061314" watchObservedRunningTime="2026-07-20 09:59:17.174583903 +0000 UTC m=+130.225513259"
Jul 20 09:59:17.389166 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.389040    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p24w6"]
Jul 20 09:59:17.389166 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.389117    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"]
Jul 20 09:59:17.525151 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:17.525001    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:18.099370 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.096297    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-6pxc9" event={"ID":"00319ed1-836b-4b3a-a2e8-da8d38bb3a66","Type":"ContainerStarted","Data":"c535eb624dbd7a6bdb238eb901c7c3cc35e6303eba106e0d3e0be5d270a1c836"}
Jul 20 09:59:18.099370 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.099062    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-6pxc9"
Jul 20 09:59:18.103295 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.102588    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc85c975b-pnwkl" event={"ID":"14d96d5e-e056-42cc-814f-0986b3a0e015","Type":"ContainerStarted","Data":"c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1"}
Jul 20 09:59:18.108965 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.108907    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-6pxc9"
Jul 20 09:59:18.110648 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.110559    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr" event={"ID":"080a463c-1129-46fd-99ac-e2c1505234a9","Type":"ContainerStarted","Data":"22f0e9a34400117660942bbb69d1a6c320f3e26f517ef0cabac2e39966b54f72"}
Jul 20 09:59:18.114648 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.114617    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x59k" event={"ID":"903c240d-9f9b-41d5-a837-95aa6feff967","Type":"ContainerStarted","Data":"0e33c4fb2e3100af221fef45b9d13b49af48513fac87ba0652e1280a06d71c23"}
Jul 20 09:59:18.119545 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.119377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp" event={"ID":"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1","Type":"ContainerStarted","Data":"afcec5d1fd69edd453541e349bf5fd6389db25b442962c49a0e6462195bc2c83"}
Jul 20 09:59:18.119545 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.119438    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp" event={"ID":"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1","Type":"ContainerStarted","Data":"0985ba63b349d803c5a02a0512103840e1105ca63b2213a79d33bc372db6ecd1"}
Jul 20 09:59:18.119545 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.119473    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp" event={"ID":"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1","Type":"ContainerStarted","Data":"f20bb442a1c111c8404b1ca26e6c5fa89afb55a0f3c1f15daa3110cd9abe4637"}
Jul 20 09:59:18.121889 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.121767    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5" event={"ID":"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd","Type":"ContainerStarted","Data":"c9cb7f049067c5d171dfc36ee92accd6e85939c5a8432913a14b707076f00860"}
Jul 20 09:59:18.125147 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.125055    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p24w6" event={"ID":"ed36d59b-3fc9-43ea-9506-f28d6955b7a3","Type":"ContainerStarted","Data":"c15c2e17f81a55bae0b6cb9a685e0a85e2958f4b349a9749897c32cde237ec40"}
Jul 20 09:59:18.131238 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.130118    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-6pxc9" podStartSLOduration=3.366180044 podStartE2EDuration="23.130098296s" totalImagesPullingTime="19.763918252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:18.128082674 +0000 UTC m=+131.179012032" watchObservedRunningTime="2026-07-20 09:59:18.130098296 +0000 UTC m=+131.181027653"
Jul 20 09:59:18.133434 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.133314    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r" event={"ID":"743df18d-8914-47c2-b93d-2245a9bd400f","Type":"ContainerStarted","Data":"cd7f1afe05d6b8a29e103c63b897a849e06587415bca50fd02291faeeb0e3b57"}
Jul 20 09:59:18.158625 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.158324    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cc85c975b-pnwkl" podStartSLOduration=5.852611567 podStartE2EDuration="19.1583031s" totalImagesPullingTime="13.305691533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:18.15764572 +0000 UTC m=+131.208575076" watchObservedRunningTime="2026-07-20 09:59:18.1583031 +0000 UTC m=+131.209232456"
Jul 20 09:59:18.948108 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:18.948075    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-687f897475-jbl2v"
Jul 20 09:59:19.104932 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:19.104740    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:19.168442 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:19.168335    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:19.657236 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:19.657197    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:59:20.159403 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:20.159364    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:20.220513 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:20.220475    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:20.749305 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:20.749158    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:20.853982 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:20.853891    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:22.933111 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:22.933065    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:23.049008 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.048970    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:23.158170 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.158131    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r" event={"ID":"743df18d-8914-47c2-b93d-2245a9bd400f","Type":"ContainerStarted","Data":"d7a5d4d40fb2e50183ebca8eb9db6ae26b6dd4890ee02943f95332db1872f2bf"}
Jul 20 09:59:23.158398 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.158288    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r" event={"ID":"743df18d-8914-47c2-b93d-2245a9bd400f","Type":"ContainerStarted","Data":"0b98aa5f16ce9246efd53c790756dbd6c89fdc050dbfd0507bd6ea38062c3c7b"}
Jul 20 09:59:23.158398 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.158326    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r" event={"ID":"743df18d-8914-47c2-b93d-2245a9bd400f","Type":"ContainerStarted","Data":"a13c81cb5d5ba0a44516ae3c5aa6111921862621ba2648c5d7e3eb14d9e54f56"}
Jul 20 09:59:23.160158 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.160126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr" event={"ID":"080a463c-1129-46fd-99ac-e2c1505234a9","Type":"ContainerStarted","Data":"a8c35104402938770a5b073fc18a1ba336df1cf945078eed2673dea962a79034"}
Jul 20 09:59:23.160323 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.160291    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:23.161953 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.161919    2555 generic.go:356] "Generic (PLEG): container finished" podID="2c315b98-9976-4328-bad4-d33913f17621" containerID="eabc2c5e919e5e932607c13e6d3597206d125e64cc8088f90e5fde604d50725b" exitCode=0
Jul 20 09:59:23.162202 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.161991    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lltk8" event={"ID":"2c315b98-9976-4328-bad4-d33913f17621","Type":"ContainerDied","Data":"eabc2c5e919e5e932607c13e6d3597206d125e64cc8088f90e5fde604d50725b"}
Jul 20 09:59:23.164868 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.164839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp" event={"ID":"1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1","Type":"ContainerStarted","Data":"935c18ad981bfd03bfb2146da59e1c418e804b2016c07d812d187fcc0f6f2096"}
Jul 20 09:59:23.166341 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.166286    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr"
Jul 20 09:59:23.167761 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.167741    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5" event={"ID":"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd","Type":"ContainerStarted","Data":"a1ed312b5adb77f82632122fd9b7a287cdb472a54625aa269eade08fcb1b42b2"}
Jul 20 09:59:23.167879 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.167795    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5" event={"ID":"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd","Type":"ContainerStarted","Data":"f33c012a569779eac07bf7918f6810633c4fb582b82b6b255e6fa5b216ddbff2"}
Jul 20 09:59:23.167879 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.167824    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5" event={"ID":"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd","Type":"ContainerStarted","Data":"794abc519d2ddd3b3ae09664da2a2afb2cf28558239990007eb894f924471929"}
Jul 20 09:59:23.169881 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.169856    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p24w6" event={"ID":"ed36d59b-3fc9-43ea-9506-f28d6955b7a3","Type":"ContainerStarted","Data":"16eba114422e7beb877c32db110144f1721cf0943b50f1e77671f59dc65adc0e"}
Jul 20 09:59:23.169975 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.169904    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p24w6" event={"ID":"ed36d59b-3fc9-43ea-9506-f28d6955b7a3","Type":"ContainerStarted","Data":"7079234b6e99cea2d9aa0dc6eeb04bb34b4d8a520f6e17e352185b0b579e3f49"}
Jul 20 09:59:23.189776 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.189717    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-njr9r" podStartSLOduration=7.036587343 podStartE2EDuration="12.189699942s" totalImagesPullingTime="5.153112599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:23.189171501 +0000 UTC m=+136.240100878" watchObservedRunningTime="2026-07-20 09:59:23.189699942 +0000 UTC m=+136.240629604"
Jul 20 09:59:23.211644 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.211585    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p24w6" podStartSLOduration=130.24183913 podStartE2EDuration="2m15.211558275s" totalImagesPullingTime="4.969719145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:23.209005854 +0000 UTC m=+136.259935206" watchObservedRunningTime="2026-07-20 09:59:23.211558275 +0000 UTC m=+136.262487631"
Jul 20 09:59:23.259694 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.259632    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-m7fmr" podStartSLOduration=3.327939137 podStartE2EDuration="8.259611755s" totalImagesPullingTime="4.931672618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:23.258724038 +0000 UTC m=+136.309653396" watchObservedRunningTime="2026-07-20 09:59:23.259611755 +0000 UTC m=+136.310541111"
Jul 20 09:59:23.326984 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.326947    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:23.389439 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.389399    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:23.861886 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.861844    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:23.862134 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.861922    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:23.867737 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.867705    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:23.892652 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:23.892551    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kqrjp" podStartSLOduration=7.860551155 podStartE2EDuration="12.892524234s" totalImagesPullingTime="5.031973079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:23.286330729 +0000 UTC m=+136.337260085" watchObservedRunningTime="2026-07-20 09:59:23.892524234 +0000 UTC m=+136.943453590"
Jul 20 09:59:24.180020 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:24.179924    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lltk8" event={"ID":"2c315b98-9976-4328-bad4-d33913f17621","Type":"ContainerStarted","Data":"d0635a75c8c66abfcbed785211be3e781ab0e48acd67fe4222e0f950772cae49"}
Jul 20 09:59:24.180614 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:24.180042    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lltk8" event={"ID":"2c315b98-9976-4328-bad4-d33913f17621","Type":"ContainerStarted","Data":"7fcf3ce8ab09104a8d7f32bc93b983bd1d2395aaf79dc0c8fc6373c3448a4bb3"}
Jul 20 09:59:24.186295 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:24.186264    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:24.224800 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:24.224748    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lltk8" podStartSLOduration=8.048094519 podStartE2EDuration="13.22472588s" totalImagesPullingTime="5.003463756s" totalInitContainerRuntime="173.167605ms" isStatefulPod=true podCreationTimestamp="2026-07-20 09:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:24.22386634 +0000 UTC m=+137.274795708" watchObservedRunningTime="2026-07-20 09:59:24.22472588 +0000 UTC m=+137.275655236"
Jul 20 09:59:24.564690 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:24.564655    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:25.188285 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:25.188225    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5" event={"ID":"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd","Type":"ContainerStarted","Data":"eb0aa0cb4d8d26edba02be86ee2e7e2b2c9046437df2c06875a13b772a18b533"}
Jul 20 09:59:25.188892 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:25.188332    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5" event={"ID":"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd","Type":"ContainerStarted","Data":"95295666009cbcad0c57adaaf05535e7dc2ea5675c3faf85eebd6fb90f4d8803"}
Jul 20 09:59:25.188892 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:25.188369    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5" event={"ID":"e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd","Type":"ContainerStarted","Data":"e8c3cfb1debd52870470a6a72ef70bc89f13c86df45e7c21da6a3237f6f0be43"}
Jul 20 09:59:25.219968 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:25.219915    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5" podStartSLOduration=4.696328352 podStartE2EDuration="11.219894722s" totalImagesPullingTime="6.52356637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:25.217117259 +0000 UTC m=+138.268046613" watchObservedRunningTime="2026-07-20 09:59:25.219894722 +0000 UTC m=+138.270824078"
Jul 20 09:59:25.445034 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:25.444935    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:25.544275 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:25.544211    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:26.055467 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:26.055426    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:26.192576 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:26.192540    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:26.932550 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:26.932514    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84f57cd5c8-nxt7q"]
Jul 20 09:59:26.995438 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:26.995396    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:27.094181 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:27.094149    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:27.197074 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:27.196988    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:27.201904 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:27.201882    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5dd6b67cdf-5wcd5"
Jul 20 09:59:27.294949 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:27.294919    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:38.004171 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:38.004128    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:38.108900 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:38.108857    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:38.284923 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:38.284824    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:38.345425 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:38.345384    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:43.164376 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.164314    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cc85c975b-pnwkl" podUID="14d96d5e-e056-42cc-814f-0986b3a0e015" containerName="console" containerID="cri-o://c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1" gracePeriod=15
Jul 20 09:59:43.434139 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.434056    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:59:43.573216 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.573178    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-oauth-config\") pod \"14d96d5e-e056-42cc-814f-0986b3a0e015\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") "
Jul 20 09:59:43.573444 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.573231    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pn5c\" (UniqueName: \"kubernetes.io/projected/14d96d5e-e056-42cc-814f-0986b3a0e015-kube-api-access-2pn5c\") pod \"14d96d5e-e056-42cc-814f-0986b3a0e015\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") "
Jul 20 09:59:43.573444 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.573316    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-oauth-serving-cert\") pod \"14d96d5e-e056-42cc-814f-0986b3a0e015\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") "
Jul 20 09:59:43.573444 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.573341    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-serving-cert\") pod \"14d96d5e-e056-42cc-814f-0986b3a0e015\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") "
Jul 20 09:59:43.573444 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.573365    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-service-ca\") pod \"14d96d5e-e056-42cc-814f-0986b3a0e015\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") "
Jul 20 09:59:43.573444 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.573408    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-console-config\") pod \"14d96d5e-e056-42cc-814f-0986b3a0e015\" (UID: \"14d96d5e-e056-42cc-814f-0986b3a0e015\") "
Jul 20 09:59:43.573881 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.573836    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-service-ca" (OuterVolumeSpecName: "service-ca") pod "14d96d5e-e056-42cc-814f-0986b3a0e015" (UID: "14d96d5e-e056-42cc-814f-0986b3a0e015"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:43.573996 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.573838    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "14d96d5e-e056-42cc-814f-0986b3a0e015" (UID: "14d96d5e-e056-42cc-814f-0986b3a0e015"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:43.573996 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.573944    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-console-config" (OuterVolumeSpecName: "console-config") pod "14d96d5e-e056-42cc-814f-0986b3a0e015" (UID: "14d96d5e-e056-42cc-814f-0986b3a0e015"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:43.575573 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.575545    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "14d96d5e-e056-42cc-814f-0986b3a0e015" (UID: "14d96d5e-e056-42cc-814f-0986b3a0e015"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:43.575675 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.575580    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "14d96d5e-e056-42cc-814f-0986b3a0e015" (UID: "14d96d5e-e056-42cc-814f-0986b3a0e015"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:43.575675 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.575620    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d96d5e-e056-42cc-814f-0986b3a0e015-kube-api-access-2pn5c" (OuterVolumeSpecName: "kube-api-access-2pn5c") pod "14d96d5e-e056-42cc-814f-0986b3a0e015" (UID: "14d96d5e-e056-42cc-814f-0986b3a0e015"). InnerVolumeSpecName "kube-api-access-2pn5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:43.673969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.673930    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-oauth-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:43.673969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.673960    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pn5c\" (UniqueName: \"kubernetes.io/projected/14d96d5e-e056-42cc-814f-0986b3a0e015-kube-api-access-2pn5c\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:43.673969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.673972    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-oauth-serving-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:43.673969 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.673981    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14d96d5e-e056-42cc-814f-0986b3a0e015-console-serving-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:43.674239 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.673991    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-service-ca\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:43.674239 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:43.674000    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14d96d5e-e056-42cc-814f-0986b3a0e015-console-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:44.261454 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:44.261424    2555 generic.go:356] "Generic (PLEG): container finished" podID="14d96d5e-e056-42cc-814f-0986b3a0e015" containerID="c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1" exitCode=0
Jul 20 09:59:44.262018 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:44.261470    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc85c975b-pnwkl"
Jul 20 09:59:44.262018 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:44.261520    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc85c975b-pnwkl" event={"ID":"14d96d5e-e056-42cc-814f-0986b3a0e015","Type":"ContainerDied","Data":"c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1"}
Jul 20 09:59:44.262018 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:44.261582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc85c975b-pnwkl" event={"ID":"14d96d5e-e056-42cc-814f-0986b3a0e015","Type":"ContainerDied","Data":"5af2a876787837ed98b82471f2ed6cc63e0d4da5fa75477f24ebc83cf704a65f"}
Jul 20 09:59:44.262018 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:44.261613    2555 scope.go:117] "RemoveContainer" containerID="c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1"
Jul 20 09:59:44.270599 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:44.270582    2555 scope.go:117] "RemoveContainer" containerID="c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1"
Jul 20 09:59:44.270880 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:59:44.270860    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1\": container with ID starting with c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1 not found: ID does not exist" containerID="c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1"
Jul 20 09:59:44.270945 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:44.270889    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1"} err="failed to get container status \"c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1\": rpc error: code = NotFound desc = could not find container \"c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1\": container with ID starting with c1576bf6b7e26101f92cac8da09c46ac6f347b20e0e5478e4ae851728836acd1 not found: ID does not exist"
Jul 20 09:59:44.284159 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:44.284129    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cc85c975b-pnwkl"]
Jul 20 09:59:44.286581 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:44.286555    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cc85c975b-pnwkl"]
Jul 20 09:59:45.490766 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:45.490734    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d96d5e-e056-42cc-814f-0986b3a0e015" path="/var/lib/kubelet/pods/14d96d5e-e056-42cc-814f-0986b3a0e015/volumes"
Jul 20 09:59:46.516642 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:46.516611    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:46.619422 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:46.619382    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:51.954142 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:51.954074    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84f57cd5c8-nxt7q" podUID="6bbe1805-14b5-4c69-94be-3e728eaea4af" containerName="console" containerID="cri-o://4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e" gracePeriod=15
Jul 20 09:59:52.260547 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.260517    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:52.294698 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.294656    2555 generic.go:356] "Generic (PLEG): container finished" podID="6bbe1805-14b5-4c69-94be-3e728eaea4af" containerID="4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e" exitCode=0
Jul 20 09:59:52.294881 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.294723    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f57cd5c8-nxt7q"
Jul 20 09:59:52.294881 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.294733    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f57cd5c8-nxt7q" event={"ID":"6bbe1805-14b5-4c69-94be-3e728eaea4af","Type":"ContainerDied","Data":"4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e"}
Jul 20 09:59:52.294881 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.294809    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f57cd5c8-nxt7q" event={"ID":"6bbe1805-14b5-4c69-94be-3e728eaea4af","Type":"ContainerDied","Data":"3409fb391cc0659c7ee038a5bd61fce41910597fca33509946cd0e854e120005"}
Jul 20 09:59:52.294881 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.294837    2555 scope.go:117] "RemoveContainer" containerID="4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e"
Jul 20 09:59:52.306838 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.306705    2555 scope.go:117] "RemoveContainer" containerID="4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e"
Jul 20 09:59:52.307425 ip-10-0-129-35 kubenswrapper[2555]: E0720 09:59:52.307346    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e\": container with ID starting with 4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e not found: ID does not exist" containerID="4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e"
Jul 20 09:59:52.307425 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.307387    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e"} err="failed to get container status \"4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e\": rpc error: code = NotFound desc = could not find container \"4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e\": container with ID starting with 4fea2cbe745a8bde606230a6b33f5ef9fb77596e103a7831cff33d985a5af17e not found: ID does not exist"
Jul 20 09:59:52.447513 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.447482    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-config\") pod \"6bbe1805-14b5-4c69-94be-3e728eaea4af\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") "
Jul 20 09:59:52.447513 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.447516    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-trusted-ca-bundle\") pod \"6bbe1805-14b5-4c69-94be-3e728eaea4af\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") "
Jul 20 09:59:52.447773 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.447556    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75znx\" (UniqueName: \"kubernetes.io/projected/6bbe1805-14b5-4c69-94be-3e728eaea4af-kube-api-access-75znx\") pod \"6bbe1805-14b5-4c69-94be-3e728eaea4af\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") "
Jul 20 09:59:52.447773 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.447589    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-oauth-config\") pod \"6bbe1805-14b5-4c69-94be-3e728eaea4af\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") "
Jul 20 09:59:52.447773 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.447669    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-oauth-serving-cert\") pod \"6bbe1805-14b5-4c69-94be-3e728eaea4af\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") "
Jul 20 09:59:52.447773 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.447697    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-serving-cert\") pod \"6bbe1805-14b5-4c69-94be-3e728eaea4af\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") "
Jul 20 09:59:52.447773 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.447733    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-service-ca\") pod \"6bbe1805-14b5-4c69-94be-3e728eaea4af\" (UID: \"6bbe1805-14b5-4c69-94be-3e728eaea4af\") "
Jul 20 09:59:52.448022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.447936    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-config" (OuterVolumeSpecName: "console-config") pod "6bbe1805-14b5-4c69-94be-3e728eaea4af" (UID: "6bbe1805-14b5-4c69-94be-3e728eaea4af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.448022 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.447966    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6bbe1805-14b5-4c69-94be-3e728eaea4af" (UID: "6bbe1805-14b5-4c69-94be-3e728eaea4af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.448128 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.448058    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6bbe1805-14b5-4c69-94be-3e728eaea4af" (UID: "6bbe1805-14b5-4c69-94be-3e728eaea4af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.448209 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.448183    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-service-ca" (OuterVolumeSpecName: "service-ca") pod "6bbe1805-14b5-4c69-94be-3e728eaea4af" (UID: "6bbe1805-14b5-4c69-94be-3e728eaea4af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.449879 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.449846    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6bbe1805-14b5-4c69-94be-3e728eaea4af" (UID: "6bbe1805-14b5-4c69-94be-3e728eaea4af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.449879 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.449863    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbe1805-14b5-4c69-94be-3e728eaea4af-kube-api-access-75znx" (OuterVolumeSpecName: "kube-api-access-75znx") pod "6bbe1805-14b5-4c69-94be-3e728eaea4af" (UID: "6bbe1805-14b5-4c69-94be-3e728eaea4af"). InnerVolumeSpecName "kube-api-access-75znx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:52.450046 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.450014    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6bbe1805-14b5-4c69-94be-3e728eaea4af" (UID: "6bbe1805-14b5-4c69-94be-3e728eaea4af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.548513 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.548439    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-oauth-serving-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.548513 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.548469    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-serving-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.548513 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.548480    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-service-ca\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.548513 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.548488    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.548513 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.548496    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bbe1805-14b5-4c69-94be-3e728eaea4af-trusted-ca-bundle\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.548513 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.548504    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75znx\" (UniqueName: \"kubernetes.io/projected/6bbe1805-14b5-4c69-94be-3e728eaea4af-kube-api-access-75znx\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.548513 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.548513    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bbe1805-14b5-4c69-94be-3e728eaea4af-console-oauth-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.639067 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.639022    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84f57cd5c8-nxt7q"]
Jul 20 09:59:52.643555 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:52.643525    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84f57cd5c8-nxt7q"]
Jul 20 09:59:53.490846 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:53.490811    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bbe1805-14b5-4c69-94be-3e728eaea4af" path="/var/lib/kubelet/pods/6bbe1805-14b5-4c69-94be-3e728eaea4af/volumes"
Jul 20 09:59:59.379372 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:59.379286    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:59.434343 ip-10-0-129-35 kubenswrapper[2555]: I0720 09:59:59.434308    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:30.452236 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:30.452191    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:30.692049 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:30.692010    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:35.550544 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.550506    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"]
Jul 20 10:00:35.550927 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.550842    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d96d5e-e056-42cc-814f-0986b3a0e015" containerName="console"
Jul 20 10:00:35.550927 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.550854    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d96d5e-e056-42cc-814f-0986b3a0e015" containerName="console"
Jul 20 10:00:35.550927 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.550873    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbe1805-14b5-4c69-94be-3e728eaea4af" containerName="console"
Jul 20 10:00:35.550927 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.550879    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbe1805-14b5-4c69-94be-3e728eaea4af" containerName="console"
Jul 20 10:00:35.551060 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.550933    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d96d5e-e056-42cc-814f-0986b3a0e015" containerName="console"
Jul 20 10:00:35.551060 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.550946    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bbe1805-14b5-4c69-94be-3e728eaea4af" containerName="console"
Jul 20 10:00:35.554148 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.554127    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.557208 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.557180    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 20 10:00:35.557332 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.557180    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 20 10:00:35.557332 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.557209    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 20 10:00:35.557332 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.557288    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8frlz\""
Jul 20 10:00:35.557511 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.557492    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 20 10:00:35.558021 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.557997    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 20 10:00:35.564343 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.564322    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 20 10:00:35.570018 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.569996    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"]
Jul 20 10:00:35.685554 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.685514    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-metrics-client-ca\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.685740 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.685560    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.685740 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.685650    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-telemeter-client-tls\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.685740 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.685694    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-federate-client-tls\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.685740 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.685713    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-secret-telemeter-client\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.685740 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.685729    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5dm\" (UniqueName: \"kubernetes.io/projected/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-kube-api-access-8z5dm\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.685896 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.685746    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.685896 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.685770    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-serving-certs-ca-bundle\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.787047 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.787006    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-metrics-client-ca\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.787176 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.787056    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.787176 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.787096    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-telemeter-client-tls\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.787176 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.787131    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-federate-client-tls\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.787176 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.787147    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-secret-telemeter-client\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.787176 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.787164    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5dm\" (UniqueName: \"kubernetes.io/projected/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-kube-api-access-8z5dm\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.787434 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.787182    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.787434 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.787206    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-serving-certs-ca-bundle\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.787937 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.787910    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-serving-certs-ca-bundle\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.788055 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.787912    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-metrics-client-ca\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.788153 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.788130    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.789954 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.789929    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.790068 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.789978    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-federate-client-tls\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.790170 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.790150    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-secret-telemeter-client\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.790232 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.790200    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-telemeter-client-tls\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.795105 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.795082    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5dm\" (UniqueName: \"kubernetes.io/projected/8f73a5b8-6d5a-4708-94d3-1c826e6894d2-kube-api-access-8z5dm\") pod \"telemeter-client-747bc6f6bd-cqkqd\" (UID: \"8f73a5b8-6d5a-4708-94d3-1c826e6894d2\") " pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:35.864567 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:35.864532    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"
Jul 20 10:00:36.005599 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:36.005572    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd"]
Jul 20 10:00:36.008005 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:00:36.007979    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f73a5b8_6d5a_4708_94d3_1c826e6894d2.slice/crio-b3a7db739b0abe6e7f2505edc441b96c9eba53f813e1790cd10c9249ac4b46a3 WatchSource:0}: Error finding container b3a7db739b0abe6e7f2505edc441b96c9eba53f813e1790cd10c9249ac4b46a3: Status 404 returned error can't find the container with id b3a7db739b0abe6e7f2505edc441b96c9eba53f813e1790cd10c9249ac4b46a3
Jul 20 10:00:36.457870 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:36.457835    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd" event={"ID":"8f73a5b8-6d5a-4708-94d3-1c826e6894d2","Type":"ContainerStarted","Data":"b3a7db739b0abe6e7f2505edc441b96c9eba53f813e1790cd10c9249ac4b46a3"}
Jul 20 10:00:38.466279 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:38.466232    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd" event={"ID":"8f73a5b8-6d5a-4708-94d3-1c826e6894d2","Type":"ContainerStarted","Data":"9ebd39d7aa973f099092f466315d4b43bf2ac1895ad335ae6930438f687fd1e4"}
Jul 20 10:00:39.472454 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:39.472418    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd" event={"ID":"8f73a5b8-6d5a-4708-94d3-1c826e6894d2","Type":"ContainerStarted","Data":"f3bf0d7f4624689194e45233a6a652a669a2e7704e0a41cfdfd09559e465808c"}
Jul 20 10:00:39.472855 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:39.472481    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd" event={"ID":"8f73a5b8-6d5a-4708-94d3-1c826e6894d2","Type":"ContainerStarted","Data":"dce8acd844af54c9c71bad74b14b0e53b5a1333d3d7f3cb3183b76bdeb99c74e"}
Jul 20 10:00:39.501073 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:39.500967    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-747bc6f6bd-cqkqd" podStartSLOduration=1.358565292 podStartE2EDuration="4.500950095s" totalImagesPullingTime="3.142384803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:00:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:00:39.499569544 +0000 UTC m=+212.550498890" watchObservedRunningTime="2026-07-20 10:00:39.500950095 +0000 UTC m=+212.551879471"
Jul 20 10:00:40.210704 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.210667    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7484895db-jgxg9"]
Jul 20 10:00:40.214055 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.214033    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.217093 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.217070    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 20 10:00:40.217199 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.217072    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 20 10:00:40.218325 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.218308    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 20 10:00:40.218475 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.218355    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 20 10:00:40.218475 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.218415    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tcc2t\""
Jul 20 10:00:40.218643 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.218624    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 20 10:00:40.224054 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.224018    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 20 10:00:40.226402 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.226377    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7484895db-jgxg9"]
Jul 20 10:00:40.261422 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.261391    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:40.327715 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.327672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-serving-cert\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.327857 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.327724    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-oauth-serving-cert\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.327857 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.327809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-oauth-config\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.327857 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.327843    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-console-config\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.327998 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.327867    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-trusted-ca-bundle\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.327998 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.327921    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-service-ca\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.327998 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.327959    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dbd\" (UniqueName: \"kubernetes.io/projected/415c0df6-ca12-443d-9728-f221e0affae8-kube-api-access-f8dbd\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.429170 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.429134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-serving-cert\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.429296 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.429177    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-oauth-serving-cert\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.429296 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.429211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-oauth-config\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.429296 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.429276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-console-config\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.429481 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.429457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-trusted-ca-bundle\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.429526 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.429507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-service-ca\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.429564 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.429537    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dbd\" (UniqueName: \"kubernetes.io/projected/415c0df6-ca12-443d-9728-f221e0affae8-kube-api-access-f8dbd\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.430134 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.430108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-console-config\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.430293 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.430155    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-oauth-serving-cert\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.430375 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.430326    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-trusted-ca-bundle\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.430375 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.430345    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-service-ca\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.431902 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.431873    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-oauth-config\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.431999 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.431972    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-serving-cert\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.438568 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.438537    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dbd\" (UniqueName: \"kubernetes.io/projected/415c0df6-ca12-443d-9728-f221e0affae8-kube-api-access-f8dbd\") pod \"console-7484895db-jgxg9\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") " pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.524617 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.524517    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:40.661833 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:40.661806    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7484895db-jgxg9"]
Jul 20 10:00:40.664408 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:00:40.664377    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415c0df6_ca12_443d_9728_f221e0affae8.slice/crio-5f86f00ef1a6929f860572f6586752cbb524e35d99cf97efb85c5e2a395306a4 WatchSource:0}: Error finding container 5f86f00ef1a6929f860572f6586752cbb524e35d99cf97efb85c5e2a395306a4: Status 404 returned error can't find the container with id 5f86f00ef1a6929f860572f6586752cbb524e35d99cf97efb85c5e2a395306a4
Jul 20 10:00:41.170518 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:41.170478    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:41.222996 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:41.222964    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:41.481782 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:41.481684    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7484895db-jgxg9" event={"ID":"415c0df6-ca12-443d-9728-f221e0affae8","Type":"ContainerStarted","Data":"8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea"}
Jul 20 10:00:41.481782 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:41.481730    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7484895db-jgxg9" event={"ID":"415c0df6-ca12-443d-9728-f221e0affae8","Type":"ContainerStarted","Data":"5f86f00ef1a6929f860572f6586752cbb524e35d99cf97efb85c5e2a395306a4"}
Jul 20 10:00:41.501944 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:41.501887    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7484895db-jgxg9" podStartSLOduration=1.501870441 podStartE2EDuration="1.501870441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:00:41.500544807 +0000 UTC m=+214.551474163" watchObservedRunningTime="2026-07-20 10:00:41.501870441 +0000 UTC m=+214.552799869"
Jul 20 10:00:50.525229 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:50.525192    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:50.525639 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:50.525271    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:50.530046 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:50.530021    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:51.523463 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:51.523433    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:00:51.698181 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:51.698121    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:51.798483 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:51.798397    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:51.854013 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:51.853977    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:59.325663 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:00:59.325620    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:04.341561 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:04.341522    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:10.750732 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:10.750698    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:10.850673 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:10.850626    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:21.988408 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:21.988366    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:22.053903 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:22.053868    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:24.116076 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:24.116039    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:45.748297 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:45.748165    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:45.806057 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:45.806019    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:53.425925 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.425891    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8578f866c6-dg7q7"]
Jul 20 10:01:53.429558 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.429533    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.445855 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.445831    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8578f866c6-dg7q7"]
Jul 20 10:01:53.492076 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.492038    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:53.546768 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.546736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-service-ca\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.546874 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.546773    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsbzn\" (UniqueName: \"kubernetes.io/projected/ef028319-63d7-41f1-b4d4-e2e70b5a1693-kube-api-access-qsbzn\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.546874 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.546844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-oauth-config\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.546940 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.546873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-oauth-serving-cert\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.546940 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.546905    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-serving-cert\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.546940 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.546930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-trusted-ca-bundle\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.547115 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.546974    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-config\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.562648 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.562617    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:53.647671 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.647641    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-serving-cert\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.647811 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.647689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-trusted-ca-bundle\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.647811 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.647743    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-config\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.647811 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.647769    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-service-ca\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.647811 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.647790    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsbzn\" (UniqueName: \"kubernetes.io/projected/ef028319-63d7-41f1-b4d4-e2e70b5a1693-kube-api-access-qsbzn\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.648002 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.647843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-oauth-config\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.648002 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.647895    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-oauth-serving-cert\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.648628 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.648600    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-config\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.648760 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.648641    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-oauth-serving-cert\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.648881 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.648852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-service-ca\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.649036 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.649019    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-trusted-ca-bundle\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.650279 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.650228    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-serving-cert\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.650372 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.650321    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-oauth-config\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.660203 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.660182    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsbzn\" (UniqueName: \"kubernetes.io/projected/ef028319-63d7-41f1-b4d4-e2e70b5a1693-kube-api-access-qsbzn\") pod \"console-8578f866c6-dg7q7\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") " pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.737902 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.737814    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:01:53.872148 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:53.872119    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8578f866c6-dg7q7"]
Jul 20 10:01:53.874812 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:01:53.874780    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef028319_63d7_41f1_b4d4_e2e70b5a1693.slice/crio-017b9d921403b70ffc665eb3ed64726e5225ab4fe901cacf4323fa03820a0c59 WatchSource:0}: Error finding container 017b9d921403b70ffc665eb3ed64726e5225ab4fe901cacf4323fa03820a0c59: Status 404 returned error can't find the container with id 017b9d921403b70ffc665eb3ed64726e5225ab4fe901cacf4323fa03820a0c59
Jul 20 10:01:54.066061 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:54.065964    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:54.123403 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:54.123368    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:54.764674 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:54.764642    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8578f866c6-dg7q7" event={"ID":"ef028319-63d7-41f1-b4d4-e2e70b5a1693","Type":"ContainerStarted","Data":"43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf"}
Jul 20 10:01:54.765087 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:54.764691    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8578f866c6-dg7q7" event={"ID":"ef028319-63d7-41f1-b4d4-e2e70b5a1693","Type":"ContainerStarted","Data":"017b9d921403b70ffc665eb3ed64726e5225ab4fe901cacf4323fa03820a0c59"}
Jul 20 10:01:54.783523 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:54.783464    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8578f866c6-dg7q7" podStartSLOduration=1.783445987 podStartE2EDuration="1.783445987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:01:54.782664695 +0000 UTC m=+287.833594066" watchObservedRunningTime="2026-07-20 10:01:54.783445987 +0000 UTC m=+287.834375409"
Jul 20 10:01:57.758726 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:57.758684    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:57.813899 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:01:57.813863    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:02:01.515358 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:01.515319    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:02:01.616847 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:01.616813    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:02:03.738216 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:03.738179    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:02:03.738648 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:03.738272    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:02:03.743385 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:03.743359    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:02:03.805283 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:03.805254    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:02:03.855273 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:03.855225    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7484895db-jgxg9"]
Jul 20 10:02:03.956998 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:03.956964    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:02:04.092912 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:04.092828    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:02:07.390942 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:07.390908    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:02:07.396829 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:07.396803    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:02:07.398504 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:07.398481    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 20 10:02:28.884006 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:28.883962    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7484895db-jgxg9" podUID="415c0df6-ca12-443d-9728-f221e0affae8" containerName="console" containerID="cri-o://8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea" gracePeriod=15
Jul 20 10:02:29.125056 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.125027    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:02:29.238397 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.238305    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8dbd\" (UniqueName: \"kubernetes.io/projected/415c0df6-ca12-443d-9728-f221e0affae8-kube-api-access-f8dbd\") pod \"415c0df6-ca12-443d-9728-f221e0affae8\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") "
Jul 20 10:02:29.238397 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.238349    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-serving-cert\") pod \"415c0df6-ca12-443d-9728-f221e0affae8\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") "
Jul 20 10:02:29.238397 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.238373    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-console-config\") pod \"415c0df6-ca12-443d-9728-f221e0affae8\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") "
Jul 20 10:02:29.238682 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.238519    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-oauth-config\") pod \"415c0df6-ca12-443d-9728-f221e0affae8\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") "
Jul 20 10:02:29.238682 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.238580    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-service-ca\") pod \"415c0df6-ca12-443d-9728-f221e0affae8\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") "
Jul 20 10:02:29.238682 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.238631    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-oauth-serving-cert\") pod \"415c0df6-ca12-443d-9728-f221e0affae8\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") "
Jul 20 10:02:29.238875 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.238691    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-trusted-ca-bundle\") pod \"415c0df6-ca12-443d-9728-f221e0affae8\" (UID: \"415c0df6-ca12-443d-9728-f221e0affae8\") "
Jul 20 10:02:29.238925 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.238809    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-console-config" (OuterVolumeSpecName: "console-config") pod "415c0df6-ca12-443d-9728-f221e0affae8" (UID: "415c0df6-ca12-443d-9728-f221e0affae8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:02:29.239001 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.238982    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-console-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:02:29.239057 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.239031    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-service-ca" (OuterVolumeSpecName: "service-ca") pod "415c0df6-ca12-443d-9728-f221e0affae8" (UID: "415c0df6-ca12-443d-9728-f221e0affae8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:02:29.239122 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.239093    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "415c0df6-ca12-443d-9728-f221e0affae8" (UID: "415c0df6-ca12-443d-9728-f221e0affae8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:02:29.239377 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.239358    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "415c0df6-ca12-443d-9728-f221e0affae8" (UID: "415c0df6-ca12-443d-9728-f221e0affae8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:02:29.240602 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.240580    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415c0df6-ca12-443d-9728-f221e0affae8-kube-api-access-f8dbd" (OuterVolumeSpecName: "kube-api-access-f8dbd") pod "415c0df6-ca12-443d-9728-f221e0affae8" (UID: "415c0df6-ca12-443d-9728-f221e0affae8"). InnerVolumeSpecName "kube-api-access-f8dbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:02:29.240976 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.240955    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "415c0df6-ca12-443d-9728-f221e0affae8" (UID: "415c0df6-ca12-443d-9728-f221e0affae8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:02:29.241041 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.240977    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "415c0df6-ca12-443d-9728-f221e0affae8" (UID: "415c0df6-ca12-443d-9728-f221e0affae8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:02:29.339822 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.339778    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-oauth-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:02:29.339822 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.339814    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-service-ca\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:02:29.339822 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.339827    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-oauth-serving-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:02:29.340071 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.339839    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/415c0df6-ca12-443d-9728-f221e0affae8-trusted-ca-bundle\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:02:29.340071 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.339853    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8dbd\" (UniqueName: \"kubernetes.io/projected/415c0df6-ca12-443d-9728-f221e0affae8-kube-api-access-f8dbd\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:02:29.340071 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.339865    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/415c0df6-ca12-443d-9728-f221e0affae8-console-serving-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:02:29.903096 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.903062    2555 generic.go:356] "Generic (PLEG): container finished" podID="415c0df6-ca12-443d-9728-f221e0affae8" containerID="8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea" exitCode=0
Jul 20 10:02:29.903527 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.903108    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7484895db-jgxg9"
Jul 20 10:02:29.903527 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.903133    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7484895db-jgxg9" event={"ID":"415c0df6-ca12-443d-9728-f221e0affae8","Type":"ContainerDied","Data":"8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea"}
Jul 20 10:02:29.903527 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.903175    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7484895db-jgxg9" event={"ID":"415c0df6-ca12-443d-9728-f221e0affae8","Type":"ContainerDied","Data":"5f86f00ef1a6929f860572f6586752cbb524e35d99cf97efb85c5e2a395306a4"}
Jul 20 10:02:29.903527 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.903195    2555 scope.go:117] "RemoveContainer" containerID="8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea"
Jul 20 10:02:29.912225 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.912202    2555 scope.go:117] "RemoveContainer" containerID="8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea"
Jul 20 10:02:29.912571 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:02:29.912550    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea\": container with ID starting with 8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea not found: ID does not exist" containerID="8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea"
Jul 20 10:02:29.912632 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.912581    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea"} err="failed to get container status \"8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea\": rpc error: code = NotFound desc = could not find container \"8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea\": container with ID starting with 8999d6145183cf9875317a1ea6fc799049ee4cadec90cef6981e166fd0d875ea not found: ID does not exist"
Jul 20 10:02:29.921191 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.921160    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7484895db-jgxg9"]
Jul 20 10:02:29.924826 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:29.924798    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7484895db-jgxg9"]
Jul 20 10:02:31.490652 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:02:31.490619    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="415c0df6-ca12-443d-9728-f221e0affae8" path="/var/lib/kubelet/pods/415c0df6-ca12-443d-9728-f221e0affae8/volumes"
Jul 20 10:03:15.625445 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:15.625411    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:17.100404 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.100367    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"]
Jul 20 10:03:17.100825 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.100682    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415c0df6-ca12-443d-9728-f221e0affae8" containerName="console"
Jul 20 10:03:17.100825 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.100693    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="415c0df6-ca12-443d-9728-f221e0affae8" containerName="console"
Jul 20 10:03:17.100825 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.100754    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="415c0df6-ca12-443d-9728-f221e0affae8" containerName="console"
Jul 20 10:03:17.104187 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.104167    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.107581 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.107548    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 20 10:03:17.107581 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.107585    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pkkmr\""
Jul 20 10:03:17.107768 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.107588    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 20 10:03:17.135865 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.135830    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"]
Jul 20 10:03:17.143512 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.143478    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbbrq\" (UniqueName: \"kubernetes.io/projected/cb399425-2d21-47c7-814b-35fb70a701c9-kube-api-access-hbbrq\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.143839 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.143531    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.143839 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.143563    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.173104 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.173065    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:17.244329 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.244285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.244523 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.244336    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.244523 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.244426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbbrq\" (UniqueName: \"kubernetes.io/projected/cb399425-2d21-47c7-814b-35fb70a701c9-kube-api-access-hbbrq\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.244757 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.244728    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.244829 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.244738    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.254758 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.254729    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbbrq\" (UniqueName: \"kubernetes.io/projected/cb399425-2d21-47c7-814b-35fb70a701c9-kube-api-access-hbbrq\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.413957 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.413906    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:17.558379 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.558348    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"]
Jul 20 10:03:17.560854 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:03:17.560822    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb399425_2d21_47c7_814b_35fb70a701c9.slice/crio-c413e2d73dea2433ed8fc0085762975341df741f67cfde58354ae69af22e2eec WatchSource:0}: Error finding container c413e2d73dea2433ed8fc0085762975341df741f67cfde58354ae69af22e2eec: Status 404 returned error can't find the container with id c413e2d73dea2433ed8fc0085762975341df741f67cfde58354ae69af22e2eec
Jul 20 10:03:17.563006 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:17.562988    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:03:18.100925 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:18.100874    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29" event={"ID":"cb399425-2d21-47c7-814b-35fb70a701c9","Type":"ContainerStarted","Data":"c413e2d73dea2433ed8fc0085762975341df741f67cfde58354ae69af22e2eec"}
Jul 20 10:03:24.134153 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:24.134116    2555 generic.go:356] "Generic (PLEG): container finished" podID="cb399425-2d21-47c7-814b-35fb70a701c9" containerID="9a6de28fa99bdd4a0d375637e16c1b92cb02af6c84f1a636e10253a1aeb036f2" exitCode=0
Jul 20 10:03:24.134572 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:24.134168    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29" event={"ID":"cb399425-2d21-47c7-814b-35fb70a701c9","Type":"ContainerDied","Data":"9a6de28fa99bdd4a0d375637e16c1b92cb02af6c84f1a636e10253a1aeb036f2"}
Jul 20 10:03:26.143885 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:26.143850    2555 generic.go:356] "Generic (PLEG): container finished" podID="cb399425-2d21-47c7-814b-35fb70a701c9" containerID="8a9d188d2a2d1c5c8db31e681633fcd039f001f9006748d8f8950b7c532f9eab" exitCode=0
Jul 20 10:03:26.144280 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:26.143928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29" event={"ID":"cb399425-2d21-47c7-814b-35fb70a701c9","Type":"ContainerDied","Data":"8a9d188d2a2d1c5c8db31e681633fcd039f001f9006748d8f8950b7c532f9eab"}
Jul 20 10:03:33.177963 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:33.177923    2555 generic.go:356] "Generic (PLEG): container finished" podID="cb399425-2d21-47c7-814b-35fb70a701c9" containerID="26c9bbd52faa481cecd13af16469109b3cdb149b474b24c3e296891af8a6dcb1" exitCode=0
Jul 20 10:03:33.178368 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:33.178005    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29" event={"ID":"cb399425-2d21-47c7-814b-35fb70a701c9","Type":"ContainerDied","Data":"26c9bbd52faa481cecd13af16469109b3cdb149b474b24c3e296891af8a6dcb1"}
Jul 20 10:03:34.309616 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:34.309586    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:34.407875 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:34.407842    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-util\") pod \"cb399425-2d21-47c7-814b-35fb70a701c9\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") "
Jul 20 10:03:34.408060 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:34.407897    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbbrq\" (UniqueName: \"kubernetes.io/projected/cb399425-2d21-47c7-814b-35fb70a701c9-kube-api-access-hbbrq\") pod \"cb399425-2d21-47c7-814b-35fb70a701c9\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") "
Jul 20 10:03:34.408060 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:34.407926    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-bundle\") pod \"cb399425-2d21-47c7-814b-35fb70a701c9\" (UID: \"cb399425-2d21-47c7-814b-35fb70a701c9\") "
Jul 20 10:03:34.408732 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:34.408705    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-bundle" (OuterVolumeSpecName: "bundle") pod "cb399425-2d21-47c7-814b-35fb70a701c9" (UID: "cb399425-2d21-47c7-814b-35fb70a701c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:03:34.410166 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:34.410136    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb399425-2d21-47c7-814b-35fb70a701c9-kube-api-access-hbbrq" (OuterVolumeSpecName: "kube-api-access-hbbrq") pod "cb399425-2d21-47c7-814b-35fb70a701c9" (UID: "cb399425-2d21-47c7-814b-35fb70a701c9"). InnerVolumeSpecName "kube-api-access-hbbrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:03:34.413153 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:34.413126    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-util" (OuterVolumeSpecName: "util") pod "cb399425-2d21-47c7-814b-35fb70a701c9" (UID: "cb399425-2d21-47c7-814b-35fb70a701c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:03:34.508927 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:34.508827    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-util\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:03:34.508927 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:34.508875    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbbrq\" (UniqueName: \"kubernetes.io/projected/cb399425-2d21-47c7-814b-35fb70a701c9-kube-api-access-hbbrq\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:03:34.508927 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:34.508886    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb399425-2d21-47c7-814b-35fb70a701c9-bundle\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:03:35.187452 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:35.187412    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29" event={"ID":"cb399425-2d21-47c7-814b-35fb70a701c9","Type":"ContainerDied","Data":"c413e2d73dea2433ed8fc0085762975341df741f67cfde58354ae69af22e2eec"}
Jul 20 10:03:35.187621 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:35.187456    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qcv29"
Jul 20 10:03:35.187621 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:35.187467    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c413e2d73dea2433ed8fc0085762975341df741f67cfde58354ae69af22e2eec"
Jul 20 10:03:38.084557 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:38.084514    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:38.182171 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:38.182135    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:43.957637 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:43.957600    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:44.287619 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.287528    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:44.305539 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.305503    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-cmql4"]
Jul 20 10:03:44.305829 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.305817    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb399425-2d21-47c7-814b-35fb70a701c9" containerName="util"
Jul 20 10:03:44.305870 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.305830    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb399425-2d21-47c7-814b-35fb70a701c9" containerName="util"
Jul 20 10:03:44.305870 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.305842    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb399425-2d21-47c7-814b-35fb70a701c9" containerName="extract"
Jul 20 10:03:44.305870 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.305848    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb399425-2d21-47c7-814b-35fb70a701c9" containerName="extract"
Jul 20 10:03:44.305870 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.305863    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb399425-2d21-47c7-814b-35fb70a701c9" containerName="pull"
Jul 20 10:03:44.305870 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.305869    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb399425-2d21-47c7-814b-35fb70a701c9" containerName="pull"
Jul 20 10:03:44.306018 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.305917    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb399425-2d21-47c7-814b-35fb70a701c9" containerName="extract"
Jul 20 10:03:44.342620 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.342579    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-cmql4"]
Jul 20 10:03:44.342777 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.342685    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:44.345913 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.345890    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 20 10:03:44.346290 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.346267    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 20 10:03:44.346414 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.346305    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 20 10:03:44.347469 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.347430    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 20 10:03:44.347582 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.347473    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-bsv9t\""
Jul 20 10:03:44.347582 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.347520    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 20 10:03:44.478411 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.478372    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:44.498517 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.498476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54t6q\" (UniqueName: \"kubernetes.io/projected/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-kube-api-access-54t6q\") pod \"keda-operator-77f648d694-cmql4\" (UID: \"e4f52391-7513-4f3c-a518-4b0e3d8e57f0\") " pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:44.498686 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.498534    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-cabundle0\") pod \"keda-operator-77f648d694-cmql4\" (UID: \"e4f52391-7513-4f3c-a518-4b0e3d8e57f0\") " pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:44.498686 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.498623    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-certificates\") pod \"keda-operator-77f648d694-cmql4\" (UID: \"e4f52391-7513-4f3c-a518-4b0e3d8e57f0\") " pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:44.534849 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.534813    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:44.591560 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.591482    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"]
Jul 20 10:03:44.599156 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.599127    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-cabundle0\") pod \"keda-operator-77f648d694-cmql4\" (UID: \"e4f52391-7513-4f3c-a518-4b0e3d8e57f0\") " pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:44.599156 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.599162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-certificates\") pod \"keda-operator-77f648d694-cmql4\" (UID: \"e4f52391-7513-4f3c-a518-4b0e3d8e57f0\") " pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:44.599377 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.599230    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54t6q\" (UniqueName: \"kubernetes.io/projected/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-kube-api-access-54t6q\") pod \"keda-operator-77f648d694-cmql4\" (UID: \"e4f52391-7513-4f3c-a518-4b0e3d8e57f0\") " pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:44.599377 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:03:44.599349    2555 secret.go:281] references non-existent secret key: ca.crt
Jul 20 10:03:44.599377 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:03:44.599370    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 20 10:03:44.599477 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:03:44.599381    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-cmql4: references non-existent secret key: ca.crt
Jul 20 10:03:44.599477 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:03:44.599460    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-certificates podName:e4f52391-7513-4f3c-a518-4b0e3d8e57f0 nodeName:}" failed. No retries permitted until 2026-07-20 10:03:45.099439256 +0000 UTC m=+398.150368608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-certificates") pod "keda-operator-77f648d694-cmql4" (UID: "e4f52391-7513-4f3c-a518-4b0e3d8e57f0") : references non-existent secret key: ca.crt
Jul 20 10:03:44.599953 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.599936    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-cabundle0\") pod \"keda-operator-77f648d694-cmql4\" (UID: \"e4f52391-7513-4f3c-a518-4b0e3d8e57f0\") " pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:44.614614 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.614585    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"]
Jul 20 10:03:44.614761 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.614710    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:44.614832 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.614806    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54t6q\" (UniqueName: \"kubernetes.io/projected/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-kube-api-access-54t6q\") pod \"keda-operator-77f648d694-cmql4\" (UID: \"e4f52391-7513-4f3c-a518-4b0e3d8e57f0\") " pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:44.617711 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.617681    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 20 10:03:44.801761 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.801717    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c8b9fcc6-e1d4-43b2-9372-b3a5643b6136-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-t94h9\" (UID: \"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:44.801969 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.801839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c8b9fcc6-e1d4-43b2-9372-b3a5643b6136-certificates\") pod \"keda-metrics-apiserver-55557df8c-t94h9\" (UID: \"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:44.801969 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.801884    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkx6w\" (UniqueName: \"kubernetes.io/projected/c8b9fcc6-e1d4-43b2-9372-b3a5643b6136-kube-api-access-gkx6w\") pod \"keda-metrics-apiserver-55557df8c-t94h9\" (UID: \"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:44.868260 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.868212    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:44.903152 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.903115    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkx6w\" (UniqueName: \"kubernetes.io/projected/c8b9fcc6-e1d4-43b2-9372-b3a5643b6136-kube-api-access-gkx6w\") pod \"keda-metrics-apiserver-55557df8c-t94h9\" (UID: \"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:44.903353 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.903170    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c8b9fcc6-e1d4-43b2-9372-b3a5643b6136-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-t94h9\" (UID: \"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:44.903353 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.903238    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c8b9fcc6-e1d4-43b2-9372-b3a5643b6136-certificates\") pod \"keda-metrics-apiserver-55557df8c-t94h9\" (UID: \"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:44.903662 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.903640    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c8b9fcc6-e1d4-43b2-9372-b3a5643b6136-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-t94h9\" (UID: \"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:44.905732 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.905705    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c8b9fcc6-e1d4-43b2-9372-b3a5643b6136-certificates\") pod \"keda-metrics-apiserver-55557df8c-t94h9\" (UID: \"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:44.919506 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.919479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkx6w\" (UniqueName: \"kubernetes.io/projected/c8b9fcc6-e1d4-43b2-9372-b3a5643b6136-kube-api-access-gkx6w\") pod \"keda-metrics-apiserver-55557df8c-t94h9\" (UID: \"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:44.931379 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:44.931354    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:45.055178 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.055125    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:45.105224 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.105179    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-certificates\") pod \"keda-operator-77f648d694-cmql4\" (UID: \"e4f52391-7513-4f3c-a518-4b0e3d8e57f0\") " pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:45.107906 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.107882    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e4f52391-7513-4f3c-a518-4b0e3d8e57f0-certificates\") pod \"keda-operator-77f648d694-cmql4\" (UID: \"e4f52391-7513-4f3c-a518-4b0e3d8e57f0\") " pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:45.111868 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.111844    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"]
Jul 20 10:03:45.114433 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:03:45.114407    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b9fcc6_e1d4_43b2_9372_b3a5643b6136.slice/crio-bcff3a8adfd2829009007eb1a1abae15a4c48178eb272b909a5ebad35e30de8c WatchSource:0}: Error finding container bcff3a8adfd2829009007eb1a1abae15a4c48178eb272b909a5ebad35e30de8c: Status 404 returned error can't find the container with id bcff3a8adfd2829009007eb1a1abae15a4c48178eb272b909a5ebad35e30de8c
Jul 20 10:03:45.143807 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.143716    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:45.229233 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.229189    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9" event={"ID":"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136","Type":"ContainerStarted","Data":"bcff3a8adfd2829009007eb1a1abae15a4c48178eb272b909a5ebad35e30de8c"}
Jul 20 10:03:45.253573 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.253533    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:45.426828 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.426728    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:45.435464 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.435440    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-cmql4"]
Jul 20 10:03:45.438112 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:03:45.438080    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f52391_7513_4f3c_a518_4b0e3d8e57f0.slice/crio-3efca474bd4466cc9a9686518d3b266ceeb6d6162fd637a8d123b2709ba2e050 WatchSource:0}: Error finding container 3efca474bd4466cc9a9686518d3b266ceeb6d6162fd637a8d123b2709ba2e050: Status 404 returned error can't find the container with id 3efca474bd4466cc9a9686518d3b266ceeb6d6162fd637a8d123b2709ba2e050
Jul 20 10:03:45.580524 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.580483    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:45.778462 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.778358    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:45.885757 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.885704    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:45.976951 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:45.976914    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:46.237637 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:46.237593    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-cmql4" event={"ID":"e4f52391-7513-4f3c-a518-4b0e3d8e57f0","Type":"ContainerStarted","Data":"3efca474bd4466cc9a9686518d3b266ceeb6d6162fd637a8d123b2709ba2e050"}
Jul 20 10:03:49.253804 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:49.253765    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-cmql4" event={"ID":"e4f52391-7513-4f3c-a518-4b0e3d8e57f0","Type":"ContainerStarted","Data":"495cd4e5bbc0ab5e3c66b5122b9e409b3c0e565025a9c1f6b5857e34e77a98dc"}
Jul 20 10:03:49.254335 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:49.253886    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:03:49.255423 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:49.255399    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9" event={"ID":"c8b9fcc6-e1d4-43b2-9372-b3a5643b6136","Type":"ContainerStarted","Data":"ab62d3d01f3f77e962ffe340591873848b00d2d452f34dae5f834e322f3e455f"}
Jul 20 10:03:49.255554 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:49.255525    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:03:49.274726 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:49.274671    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-cmql4" podStartSLOduration=1.935650637 podStartE2EDuration="5.274655149s" totalImagesPullingTime="3.339004512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:03:49.274021364 +0000 UTC m=+402.324950718" watchObservedRunningTime="2026-07-20 10:03:49.274655149 +0000 UTC m=+402.325584504"
Jul 20 10:03:49.298976 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:03:49.298920    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9" podStartSLOduration=1.640970042 podStartE2EDuration="5.298905166s" totalImagesPullingTime="3.657935124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:03:49.297456211 +0000 UTC m=+402.348385567" watchObservedRunningTime="2026-07-20 10:03:49.298905166 +0000 UTC m=+402.349834571"
Jul 20 10:04:00.263932 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:00.263900    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-t94h9"
Jul 20 10:04:10.261878 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:10.261846    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-cmql4"
Jul 20 10:04:22.715946 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:22.715909    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:22.815094 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:22.815062    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:56.546032 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.545995    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:56.684463 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.684431    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-v4nj6"]
Jul 20 10:04:56.687930 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.687900    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-v4nj6"
Jul 20 10:04:56.692437 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.692393    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 20 10:04:56.692811 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.692787    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zk8mq\""
Jul 20 10:04:56.692952 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.692883    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 20 10:04:56.693068 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.693057    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 20 10:04:56.705107 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.705080    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-v4nj6"]
Jul 20 10:04:56.715815 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.715784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhs2l\" (UniqueName: \"kubernetes.io/projected/81cb20cb-8c0d-45a3-a410-2afb50bdbe8e-kube-api-access-xhs2l\") pod \"seaweedfs-64568bcd49-v4nj6\" (UID: \"81cb20cb-8c0d-45a3-a410-2afb50bdbe8e\") " pod="kserve/seaweedfs-64568bcd49-v4nj6"
Jul 20 10:04:56.715970 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.715887    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/81cb20cb-8c0d-45a3-a410-2afb50bdbe8e-data\") pod \"seaweedfs-64568bcd49-v4nj6\" (UID: \"81cb20cb-8c0d-45a3-a410-2afb50bdbe8e\") " pod="kserve/seaweedfs-64568bcd49-v4nj6"
Jul 20 10:04:56.790879 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.790838    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:56.816691 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.816595    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/81cb20cb-8c0d-45a3-a410-2afb50bdbe8e-data\") pod \"seaweedfs-64568bcd49-v4nj6\" (UID: \"81cb20cb-8c0d-45a3-a410-2afb50bdbe8e\") " pod="kserve/seaweedfs-64568bcd49-v4nj6"
Jul 20 10:04:56.816691 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.816650    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhs2l\" (UniqueName: \"kubernetes.io/projected/81cb20cb-8c0d-45a3-a410-2afb50bdbe8e-kube-api-access-xhs2l\") pod \"seaweedfs-64568bcd49-v4nj6\" (UID: \"81cb20cb-8c0d-45a3-a410-2afb50bdbe8e\") " pod="kserve/seaweedfs-64568bcd49-v4nj6"
Jul 20 10:04:56.816987 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.816965    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/81cb20cb-8c0d-45a3-a410-2afb50bdbe8e-data\") pod \"seaweedfs-64568bcd49-v4nj6\" (UID: \"81cb20cb-8c0d-45a3-a410-2afb50bdbe8e\") " pod="kserve/seaweedfs-64568bcd49-v4nj6"
Jul 20 10:04:56.829769 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.829731    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhs2l\" (UniqueName: \"kubernetes.io/projected/81cb20cb-8c0d-45a3-a410-2afb50bdbe8e-kube-api-access-xhs2l\") pod \"seaweedfs-64568bcd49-v4nj6\" (UID: \"81cb20cb-8c0d-45a3-a410-2afb50bdbe8e\") " pod="kserve/seaweedfs-64568bcd49-v4nj6"
Jul 20 10:04:56.853594 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.853555    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:56.999643 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:56.999611    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-v4nj6"
Jul 20 10:04:57.137706 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:57.137675    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-v4nj6"]
Jul 20 10:04:57.141571 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:04:57.141530    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cb20cb_8c0d_45a3_a410_2afb50bdbe8e.slice/crio-cb48fc5624a5131f740a6d361c7f09b587e1b5348c78edf80102f9e931c40006 WatchSource:0}: Error finding container cb48fc5624a5131f740a6d361c7f09b587e1b5348c78edf80102f9e931c40006: Status 404 returned error can't find the container with id cb48fc5624a5131f740a6d361c7f09b587e1b5348c78edf80102f9e931c40006
Jul 20 10:04:57.193259 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:57.193205    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:57.531544 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:57.531502    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-v4nj6" event={"ID":"81cb20cb-8c0d-45a3-a410-2afb50bdbe8e","Type":"ContainerStarted","Data":"cb48fc5624a5131f740a6d361c7f09b587e1b5348c78edf80102f9e931c40006"}
Jul 20 10:04:58.099282 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:58.099203    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:58.206117 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:58.206080    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:58.300867 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:04:58.300806    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:00.546410 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:00.546377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-v4nj6" event={"ID":"81cb20cb-8c0d-45a3-a410-2afb50bdbe8e","Type":"ContainerStarted","Data":"d0e41f2cc7ca7a4fcb5b747b96a13a6fd07fc98d49ec66db379d84a5a56c64d6"}
Jul 20 10:05:00.546799 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:00.546455    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-v4nj6"
Jul 20 10:05:00.565454 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:00.565397    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-v4nj6" podStartSLOduration=1.734065721 podStartE2EDuration="4.565379509s" totalImagesPullingTime="2.831313788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:05:00.56401725 +0000 UTC m=+473.614946606" watchObservedRunningTime="2026-07-20 10:05:00.565379509 +0000 UTC m=+473.616308864"
Jul 20 10:05:06.553823 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:06.553788    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-v4nj6"
Jul 20 10:05:17.801628 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:17.801589    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:17.861861 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:17.861820    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:19.941561 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:19.941521    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:20.042632 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:20.042598    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:52.139499 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:52.139459    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:54.122180 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.122145    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-595fc7c78d-qpn9d"]
Jul 20 10:05:54.127353 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.127330    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.166111 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.166074    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595fc7c78d-qpn9d"]
Jul 20 10:05:54.213565 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.213519    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-console-config\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.213565 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.213559    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-trusted-ca-bundle\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.213828 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.213609    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7944cd0f-9a9f-4449-85a3-983ea9036841-console-serving-cert\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.213828 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.213655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-service-ca\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.213828 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.213681    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-oauth-serving-cert\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.213828 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.213732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7944cd0f-9a9f-4449-85a3-983ea9036841-console-oauth-config\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.213828 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.213764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mztrh\" (UniqueName: \"kubernetes.io/projected/7944cd0f-9a9f-4449-85a3-983ea9036841-kube-api-access-mztrh\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.242194 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.242161    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:54.314580 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.314543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-console-config\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.314580 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.314581    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-trusted-ca-bundle\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.314778 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.314629    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7944cd0f-9a9f-4449-85a3-983ea9036841-console-serving-cert\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.314778 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.314659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-service-ca\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.314778 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.314683    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-oauth-serving-cert\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.314778 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.314742    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7944cd0f-9a9f-4449-85a3-983ea9036841-console-oauth-config\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.314961 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.314776    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mztrh\" (UniqueName: \"kubernetes.io/projected/7944cd0f-9a9f-4449-85a3-983ea9036841-kube-api-access-mztrh\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.315513 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.315447    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-service-ca\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.315513 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.315486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-console-config\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.315679 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.315520    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-oauth-serving-cert\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.315716 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.315684    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7944cd0f-9a9f-4449-85a3-983ea9036841-trusted-ca-bundle\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.317113 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.317090    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7944cd0f-9a9f-4449-85a3-983ea9036841-console-oauth-config\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.317270 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.317234    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7944cd0f-9a9f-4449-85a3-983ea9036841-console-serving-cert\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.325967 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.325939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mztrh\" (UniqueName: \"kubernetes.io/projected/7944cd0f-9a9f-4449-85a3-983ea9036841-kube-api-access-mztrh\") pod \"console-595fc7c78d-qpn9d\" (UID: \"7944cd0f-9a9f-4449-85a3-983ea9036841\") " pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.355011 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.354976    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:54.436216 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.436123    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:05:54.593138 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.593112    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595fc7c78d-qpn9d"]
Jul 20 10:05:54.595635 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:05:54.595605    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7944cd0f_9a9f_4449_85a3_983ea9036841.slice/crio-fe30e0f9873291ba8f4dbed98d43f255c299f9cd704c2b9832d18ad99b5cd5a6 WatchSource:0}: Error finding container fe30e0f9873291ba8f4dbed98d43f255c299f9cd704c2b9832d18ad99b5cd5a6: Status 404 returned error can't find the container with id fe30e0f9873291ba8f4dbed98d43f255c299f9cd704c2b9832d18ad99b5cd5a6
Jul 20 10:05:54.773319 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.773205    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595fc7c78d-qpn9d" event={"ID":"7944cd0f-9a9f-4449-85a3-983ea9036841","Type":"ContainerStarted","Data":"7b4b3e86d1da90f18fe9e6f09dd84d8d3a9b5352802d3ac61584246430ea76bd"}
Jul 20 10:05:54.773319 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.773317    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595fc7c78d-qpn9d" event={"ID":"7944cd0f-9a9f-4449-85a3-983ea9036841","Type":"ContainerStarted","Data":"fe30e0f9873291ba8f4dbed98d43f255c299f9cd704c2b9832d18ad99b5cd5a6"}
Jul 20 10:05:54.804345 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:54.804282    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-595fc7c78d-qpn9d" podStartSLOduration=0.804264249 podStartE2EDuration="804.264249ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:05:54.802935783 +0000 UTC m=+527.853865139" watchObservedRunningTime="2026-07-20 10:05:54.804264249 +0000 UTC m=+527.855193605"
Jul 20 10:05:56.757546 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:05:56.757503    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:04.436510 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:04.436398    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:06:04.436510 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:04.436482    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:06:04.441713 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:04.441684    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:06:04.819427 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:04.819348    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-595fc7c78d-qpn9d"
Jul 20 10:06:04.907062 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:04.907029    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8578f866c6-dg7q7"]
Jul 20 10:06:05.109726 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:05.109689    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:05.205084 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:05.205052    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:05.282731 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:05.282693    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:10.980754 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:10.980707    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:11.044059 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:11.044026    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:21.489237 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.489194    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:21.540215 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.540180    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-wpj4f"]
Jul 20 10:06:21.543881 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.543856    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:06:21.546654 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.546630    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 20 10:06:21.546777 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.546664    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-kjjss\""
Jul 20 10:06:21.557406 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.557379    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-cr6cj"]
Jul 20 10:06:21.560545 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.560525    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wpj4f"]
Jul 20 10:06:21.560666 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.560609    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:21.563552 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.563534    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-gwsft\""
Jul 20 10:06:21.563669 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.563586    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 20 10:06:21.568615 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.568375    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:21.570861 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.570834    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-cr6cj"]
Jul 20 10:06:21.641073 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.641033    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef21c342-5785-4b06-84cb-89a99bc7083f-cert\") pod \"odh-model-controller-84b54b7f97-cr6cj\" (UID: \"ef21c342-5785-4b06-84cb-89a99bc7083f\") " pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:21.641270 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.641123    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df7tp\" (UniqueName: \"kubernetes.io/projected/cf9c9e27-5b07-455e-bc14-9d514d052725-kube-api-access-df7tp\") pod \"model-serving-api-fd65d7d6-wpj4f\" (UID: \"cf9c9e27-5b07-455e-bc14-9d514d052725\") " pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:06:21.641270 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.641146    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx84d\" (UniqueName: \"kubernetes.io/projected/ef21c342-5785-4b06-84cb-89a99bc7083f-kube-api-access-lx84d\") pod \"odh-model-controller-84b54b7f97-cr6cj\" (UID: \"ef21c342-5785-4b06-84cb-89a99bc7083f\") " pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:21.641270 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.641166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf9c9e27-5b07-455e-bc14-9d514d052725-tls-certs\") pod \"model-serving-api-fd65d7d6-wpj4f\" (UID: \"cf9c9e27-5b07-455e-bc14-9d514d052725\") " pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:06:21.741787 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.741675    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf9c9e27-5b07-455e-bc14-9d514d052725-tls-certs\") pod \"model-serving-api-fd65d7d6-wpj4f\" (UID: \"cf9c9e27-5b07-455e-bc14-9d514d052725\") " pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:06:21.741787 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.741757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef21c342-5785-4b06-84cb-89a99bc7083f-cert\") pod \"odh-model-controller-84b54b7f97-cr6cj\" (UID: \"ef21c342-5785-4b06-84cb-89a99bc7083f\") " pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:21.742042 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:06:21.741790    2555 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 20 10:06:21.742042 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.741853    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df7tp\" (UniqueName: \"kubernetes.io/projected/cf9c9e27-5b07-455e-bc14-9d514d052725-kube-api-access-df7tp\") pod \"model-serving-api-fd65d7d6-wpj4f\" (UID: \"cf9c9e27-5b07-455e-bc14-9d514d052725\") " pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:06:21.742042 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:06:21.741867    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9c9e27-5b07-455e-bc14-9d514d052725-tls-certs podName:cf9c9e27-5b07-455e-bc14-9d514d052725 nodeName:}" failed. No retries permitted until 2026-07-20 10:06:22.241850153 +0000 UTC m=+555.292779485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cf9c9e27-5b07-455e-bc14-9d514d052725-tls-certs") pod "model-serving-api-fd65d7d6-wpj4f" (UID: "cf9c9e27-5b07-455e-bc14-9d514d052725") : secret "model-serving-api-tls" not found
Jul 20 10:06:21.742042 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:06:21.741878    2555 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 20 10:06:21.742042 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.741900    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx84d\" (UniqueName: \"kubernetes.io/projected/ef21c342-5785-4b06-84cb-89a99bc7083f-kube-api-access-lx84d\") pod \"odh-model-controller-84b54b7f97-cr6cj\" (UID: \"ef21c342-5785-4b06-84cb-89a99bc7083f\") " pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:21.742042 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:06:21.741932    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef21c342-5785-4b06-84cb-89a99bc7083f-cert podName:ef21c342-5785-4b06-84cb-89a99bc7083f nodeName:}" failed. No retries permitted until 2026-07-20 10:06:22.241919757 +0000 UTC m=+555.292849094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ef21c342-5785-4b06-84cb-89a99bc7083f-cert") pod "odh-model-controller-84b54b7f97-cr6cj" (UID: "ef21c342-5785-4b06-84cb-89a99bc7083f") : secret "odh-model-controller-webhook-cert" not found
Jul 20 10:06:21.761294 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.761260    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df7tp\" (UniqueName: \"kubernetes.io/projected/cf9c9e27-5b07-455e-bc14-9d514d052725-kube-api-access-df7tp\") pod \"model-serving-api-fd65d7d6-wpj4f\" (UID: \"cf9c9e27-5b07-455e-bc14-9d514d052725\") " pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:06:21.762733 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.762707    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx84d\" (UniqueName: \"kubernetes.io/projected/ef21c342-5785-4b06-84cb-89a99bc7083f-kube-api-access-lx84d\") pod \"odh-model-controller-84b54b7f97-cr6cj\" (UID: \"ef21c342-5785-4b06-84cb-89a99bc7083f\") " pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:21.802878 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.802832    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:21.917545 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:21.917510    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:22.246193 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.246155    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef21c342-5785-4b06-84cb-89a99bc7083f-cert\") pod \"odh-model-controller-84b54b7f97-cr6cj\" (UID: \"ef21c342-5785-4b06-84cb-89a99bc7083f\") " pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:22.246423 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.246276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf9c9e27-5b07-455e-bc14-9d514d052725-tls-certs\") pod \"model-serving-api-fd65d7d6-wpj4f\" (UID: \"cf9c9e27-5b07-455e-bc14-9d514d052725\") " pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:06:22.248668 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.248634    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf9c9e27-5b07-455e-bc14-9d514d052725-tls-certs\") pod \"model-serving-api-fd65d7d6-wpj4f\" (UID: \"cf9c9e27-5b07-455e-bc14-9d514d052725\") " pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:06:22.248668 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.248656    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef21c342-5785-4b06-84cb-89a99bc7083f-cert\") pod \"odh-model-controller-84b54b7f97-cr6cj\" (UID: \"ef21c342-5785-4b06-84cb-89a99bc7083f\") " pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:22.454453 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.454417    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:06:22.471843 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.471820    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:22.618027 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.617859    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wpj4f"]
Jul 20 10:06:22.620793 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:06:22.620762    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9c9e27_5b07_455e_bc14_9d514d052725.slice/crio-6e339d4069572eee32aab41f5af401dcb4a770b6e28b39ed3ec84cc403c8a5ae WatchSource:0}: Error finding container 6e339d4069572eee32aab41f5af401dcb4a770b6e28b39ed3ec84cc403c8a5ae: Status 404 returned error can't find the container with id 6e339d4069572eee32aab41f5af401dcb4a770b6e28b39ed3ec84cc403c8a5ae
Jul 20 10:06:22.645686 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.645651    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-cr6cj"]
Jul 20 10:06:22.647087 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.646924    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:22.649230 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:06:22.649196    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef21c342_5785_4b06_84cb_89a99bc7083f.slice/crio-11aa11de88d47cb4f044433300e852cee041ab0dfeb4d183574dd0bc6cd8e3ce WatchSource:0}: Error finding container 11aa11de88d47cb4f044433300e852cee041ab0dfeb4d183574dd0bc6cd8e3ce: Status 404 returned error can't find the container with id 11aa11de88d47cb4f044433300e852cee041ab0dfeb4d183574dd0bc6cd8e3ce
Jul 20 10:06:22.775989 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.775899    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:22.892750 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.892712    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wpj4f" event={"ID":"cf9c9e27-5b07-455e-bc14-9d514d052725","Type":"ContainerStarted","Data":"6e339d4069572eee32aab41f5af401dcb4a770b6e28b39ed3ec84cc403c8a5ae"}
Jul 20 10:06:22.893968 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:22.893935    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-cr6cj" event={"ID":"ef21c342-5785-4b06-84cb-89a99bc7083f","Type":"ContainerStarted","Data":"11aa11de88d47cb4f044433300e852cee041ab0dfeb4d183574dd0bc6cd8e3ce"}
Jul 20 10:06:26.919410 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:26.919365    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wpj4f" event={"ID":"cf9c9e27-5b07-455e-bc14-9d514d052725","Type":"ContainerStarted","Data":"a3918432d7e0c2704e8017c241ad69188a2e1e0b1c94fd6a276864cea8de09ab"}
Jul 20 10:06:26.919902 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:26.919493    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:06:26.920934 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:26.920909    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-cr6cj" event={"ID":"ef21c342-5785-4b06-84cb-89a99bc7083f","Type":"ContainerStarted","Data":"162d0ee89c7127c4d8733ae97756ddc27f2f84067fcf61057aee805ffe134ee1"}
Jul 20 10:06:26.921064 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:26.920969    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:26.950406 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:26.950350    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-wpj4f" podStartSLOduration=2.152905941 podStartE2EDuration="5.950333243s" totalImagesPullingTime="3.797427302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:26.949854489 +0000 UTC m=+560.000783844" watchObservedRunningTime="2026-07-20 10:06:26.950333243 +0000 UTC m=+560.001262598"
Jul 20 10:06:29.928601 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:29.928523    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8578f866c6-dg7q7" podUID="ef028319-63d7-41f1-b4d4-e2e70b5a1693" containerName="console" containerID="cri-o://43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf" gracePeriod=15
Jul 20 10:06:30.185625 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.185563    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:06:30.206393 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.206340    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-cr6cj" podStartSLOduration=5.491288056 podStartE2EDuration="9.206323865s" totalImagesPullingTime="3.715035809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:26.983607839 +0000 UTC m=+560.034537194" watchObservedRunningTime="2026-07-20 10:06:30.206323865 +0000 UTC m=+563.257253219"
Jul 20 10:06:30.324624 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.324586    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-oauth-config\") pod \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") "
Jul 20 10:06:30.324624 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.324627    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsbzn\" (UniqueName: \"kubernetes.io/projected/ef028319-63d7-41f1-b4d4-e2e70b5a1693-kube-api-access-qsbzn\") pod \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") "
Jul 20 10:06:30.324850 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.324650    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-config\") pod \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") "
Jul 20 10:06:30.324850 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.324700    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-serving-cert\") pod \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") "
Jul 20 10:06:30.324850 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.324737    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-oauth-serving-cert\") pod \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") "
Jul 20 10:06:30.324850 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.324759    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-trusted-ca-bundle\") pod \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") "
Jul 20 10:06:30.325035 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.324923    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-service-ca\") pod \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\" (UID: \"ef028319-63d7-41f1-b4d4-e2e70b5a1693\") "
Jul 20 10:06:30.325165 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.325128    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-config" (OuterVolumeSpecName: "console-config") pod "ef028319-63d7-41f1-b4d4-e2e70b5a1693" (UID: "ef028319-63d7-41f1-b4d4-e2e70b5a1693"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:06:30.325325 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.325296    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:06:30.325325 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.325309    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ef028319-63d7-41f1-b4d4-e2e70b5a1693" (UID: "ef028319-63d7-41f1-b4d4-e2e70b5a1693"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:06:30.325457 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.325336    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ef028319-63d7-41f1-b4d4-e2e70b5a1693" (UID: "ef028319-63d7-41f1-b4d4-e2e70b5a1693"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:06:30.325534 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.325508    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-service-ca" (OuterVolumeSpecName: "service-ca") pod "ef028319-63d7-41f1-b4d4-e2e70b5a1693" (UID: "ef028319-63d7-41f1-b4d4-e2e70b5a1693"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:06:30.326960 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.326931    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef028319-63d7-41f1-b4d4-e2e70b5a1693-kube-api-access-qsbzn" (OuterVolumeSpecName: "kube-api-access-qsbzn") pod "ef028319-63d7-41f1-b4d4-e2e70b5a1693" (UID: "ef028319-63d7-41f1-b4d4-e2e70b5a1693"). InnerVolumeSpecName "kube-api-access-qsbzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:06:30.327314 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.327292    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ef028319-63d7-41f1-b4d4-e2e70b5a1693" (UID: "ef028319-63d7-41f1-b4d4-e2e70b5a1693"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:06:30.327419 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.327318    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ef028319-63d7-41f1-b4d4-e2e70b5a1693" (UID: "ef028319-63d7-41f1-b4d4-e2e70b5a1693"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:06:30.426280 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.426223    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-serving-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:06:30.426280 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.426274    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-oauth-serving-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:06:30.426280 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.426283    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-trusted-ca-bundle\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:06:30.426280 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.426293    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef028319-63d7-41f1-b4d4-e2e70b5a1693-service-ca\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:06:30.426550 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.426301    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef028319-63d7-41f1-b4d4-e2e70b5a1693-console-oauth-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:06:30.426550 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.426309    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsbzn\" (UniqueName: \"kubernetes.io/projected/ef028319-63d7-41f1-b4d4-e2e70b5a1693-kube-api-access-qsbzn\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:06:30.938302 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.938268    2555 generic.go:356] "Generic (PLEG): container finished" podID="ef028319-63d7-41f1-b4d4-e2e70b5a1693" containerID="43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf" exitCode=0
Jul 20 10:06:30.938302 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.938304    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8578f866c6-dg7q7"
Jul 20 10:06:30.938769 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.938331    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8578f866c6-dg7q7" event={"ID":"ef028319-63d7-41f1-b4d4-e2e70b5a1693","Type":"ContainerDied","Data":"43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf"}
Jul 20 10:06:30.938769 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.938382    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8578f866c6-dg7q7" event={"ID":"ef028319-63d7-41f1-b4d4-e2e70b5a1693","Type":"ContainerDied","Data":"017b9d921403b70ffc665eb3ed64726e5225ab4fe901cacf4323fa03820a0c59"}
Jul 20 10:06:30.938769 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.938402    2555 scope.go:117] "RemoveContainer" containerID="43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf"
Jul 20 10:06:30.947349 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.947330    2555 scope.go:117] "RemoveContainer" containerID="43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf"
Jul 20 10:06:30.947644 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:06:30.947623    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf\": container with ID starting with 43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf not found: ID does not exist" containerID="43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf"
Jul 20 10:06:30.947691 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.947654    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf"} err="failed to get container status \"43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf\": rpc error: code = NotFound desc = could not find container \"43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf\": container with ID starting with 43ccc503a9141b8617bb0077aa9fc777bab7e26039a6432351a59f80ae157ddf not found: ID does not exist"
Jul 20 10:06:30.962430 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.962396    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8578f866c6-dg7q7"]
Jul 20 10:06:30.965289 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:30.965261    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8578f866c6-dg7q7"]
Jul 20 10:06:31.490952 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:31.490918    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef028319-63d7-41f1-b4d4-e2e70b5a1693" path="/var/lib/kubelet/pods/ef028319-63d7-41f1-b4d4-e2e70b5a1693/volumes"
Jul 20 10:06:37.927870 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:37.927839    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-cr6cj"
Jul 20 10:06:37.929725 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:06:37.929699    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-wpj4f"
Jul 20 10:07:07.423872 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:07:07.423837    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:07:07.427338 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:07:07.427311    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:07:57.274688 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:07:57.274609    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:57.362800 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:07:57.362762    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:08:09.848297 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:09.848233    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:08:10.000527 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.000490    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"]
Jul 20 10:08:10.000850 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.000839    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef028319-63d7-41f1-b4d4-e2e70b5a1693" containerName="console"
Jul 20 10:08:10.000891 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.000852    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef028319-63d7-41f1-b4d4-e2e70b5a1693" containerName="console"
Jul 20 10:08:10.000929 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.000906    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef028319-63d7-41f1-b4d4-e2e70b5a1693" containerName="console"
Jul 20 10:08:10.004128 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.004111    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.009623 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.009588    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a9709-predictor-serving-cert\""
Jul 20 10:08:10.009877 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.009856    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:08:10.010022 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.009890    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:08:10.010022 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.009931    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jbsdb\""
Jul 20 10:08:10.010022 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.009931    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:08:10.010022 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.010013    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a9709-kube-rbac-proxy-sar-config\""
Jul 20 10:08:10.010293 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.010016    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:08:10.036292 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.036259    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"]
Jul 20 10:08:10.073386 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.073351    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.073386 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.073399    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzpz2\" (UniqueName: \"kubernetes.io/projected/32dba575-515b-4185-bb26-4e4a59694f1d-kube-api-access-xzpz2\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.073616 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.073434    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32dba575-515b-4185-bb26-4e4a59694f1d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.073616 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.073524    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32dba575-515b-4185-bb26-4e4a59694f1d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.073616 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.073573    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.135947 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.135916    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:08:10.174803 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.174768    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32dba575-515b-4185-bb26-4e4a59694f1d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.175015 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.174851    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32dba575-515b-4185-bb26-4e4a59694f1d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.175015 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.174895    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.175015 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.174996    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.175177 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.175043    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzpz2\" (UniqueName: \"kubernetes.io/projected/32dba575-515b-4185-bb26-4e4a59694f1d-kube-api-access-xzpz2\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.175449 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.175405    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32dba575-515b-4185-bb26-4e4a59694f1d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.175884 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.175855    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.175978 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.175904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.177779 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.177756    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32dba575-515b-4185-bb26-4e4a59694f1d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.184434 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.184407    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzpz2\" (UniqueName: \"kubernetes.io/projected/32dba575-515b-4185-bb26-4e4a59694f1d-kube-api-access-xzpz2\") pod \"isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.314683 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.314635    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:10.458139 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:10.458099    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"]
Jul 20 10:08:10.462091 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:08:10.462060    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32dba575_515b_4185_bb26_4e4a59694f1d.slice/crio-24b3bd87c7a676e323dbc5edd3b26296193f3ceb973cef95e68b0cfcb028f6ee WatchSource:0}: Error finding container 24b3bd87c7a676e323dbc5edd3b26296193f3ceb973cef95e68b0cfcb028f6ee: Status 404 returned error can't find the container with id 24b3bd87c7a676e323dbc5edd3b26296193f3ceb973cef95e68b0cfcb028f6ee
Jul 20 10:08:11.372125 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:11.371915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" event={"ID":"32dba575-515b-4185-bb26-4e4a59694f1d","Type":"ContainerStarted","Data":"24b3bd87c7a676e323dbc5edd3b26296193f3ceb973cef95e68b0cfcb028f6ee"}
Jul 20 10:08:14.396462 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:14.396418    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" event={"ID":"32dba575-515b-4185-bb26-4e4a59694f1d","Type":"ContainerStarted","Data":"86a42f804a84fc87405021da39f92a03d9fc5edd40021f78a471d5ce21de692b"}
Jul 20 10:08:18.415339 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:18.415305    2555 generic.go:356] "Generic (PLEG): container finished" podID="32dba575-515b-4185-bb26-4e4a59694f1d" containerID="86a42f804a84fc87405021da39f92a03d9fc5edd40021f78a471d5ce21de692b" exitCode=0
Jul 20 10:08:18.415746 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:18.415385    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" event={"ID":"32dba575-515b-4185-bb26-4e4a59694f1d","Type":"ContainerDied","Data":"86a42f804a84fc87405021da39f92a03d9fc5edd40021f78a471d5ce21de692b"}
Jul 20 10:08:18.416620 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:18.416605    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:08:33.518634 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:33.518594    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" event={"ID":"32dba575-515b-4185-bb26-4e4a59694f1d","Type":"ContainerStarted","Data":"d1c7de8457aeba48aeb224bf38f5a1565219dc0f3e06ba9da556c3cbdd57e1c8"}
Jul 20 10:08:35.532547 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:35.532507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" event={"ID":"32dba575-515b-4185-bb26-4e4a59694f1d","Type":"ContainerStarted","Data":"b37831f482e7b1b9a6544936cc3f2f3f95f0c3836e4cab3da424c163ccf9de6a"}
Jul 20 10:08:37.544803 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:37.544761    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" event={"ID":"32dba575-515b-4185-bb26-4e4a59694f1d","Type":"ContainerStarted","Data":"132394e79261ad7f9e89fd36debd39cc16572c2e2abc6472d774512cc6861cbc"}
Jul 20 10:08:37.545330 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:37.544999    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:37.545330 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:37.545048    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:37.545330 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:37.545084    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:37.546839 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:37.546793    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:08:37.547568 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:37.547543    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:08:37.572534 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:37.572481    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podStartSLOduration=3.132635559 podStartE2EDuration="28.572466676s" totalImagesPullingTime="21.613203163s" totalInitContainerRuntime="3.826627954s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:08:09 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:08:37.570896476 +0000 UTC m=+690.621825832" watchObservedRunningTime="2026-07-20 10:08:37.572466676 +0000 UTC m=+690.623396030"
Jul 20 10:08:38.549108 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:38.549064    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:08:38.549664 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:38.549638    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:08:43.553820 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:43.553790    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:08:43.554410 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:43.554381    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:08:43.554803 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:43.554778    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:08:53.555378 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:53.555333    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:08:53.555874 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:08:53.555832    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:09:03.555015 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:03.554918    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:09:03.555481 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:03.555454    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:09:13.554966 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:13.554915    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:09:13.555418 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:13.555310    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:09:23.554937 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:23.554893    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:09:23.555443 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:23.555413    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:09:33.554560 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:33.554515    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:09:33.555046 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:33.554972    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:09:43.554506 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:43.554465    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:09:43.555014 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:43.554988    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:09:53.555015 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:53.554980    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:09:53.555646 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:09:53.555511    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:10:04.987157 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:04.987114    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"]
Jul 20 10:10:04.987550 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:04.987505    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" containerID="cri-o://d1c7de8457aeba48aeb224bf38f5a1565219dc0f3e06ba9da556c3cbdd57e1c8" gracePeriod=30
Jul 20 10:10:04.987630 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:04.987598    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kube-rbac-proxy" containerID="cri-o://b37831f482e7b1b9a6544936cc3f2f3f95f0c3836e4cab3da424c163ccf9de6a" gracePeriod=30
Jul 20 10:10:04.987748 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:04.987599    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" containerID="cri-o://132394e79261ad7f9e89fd36debd39cc16572c2e2abc6472d774512cc6861cbc" gracePeriod=30
Jul 20 10:10:05.036691 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.036617    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:05.087889 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.087858    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"]
Jul 20 10:10:05.091790 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.091765    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.096519 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.096490    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-d73f8-kube-rbac-proxy-sar-config\""
Jul 20 10:10:05.096667 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.096533    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-d73f8-predictor-serving-cert\""
Jul 20 10:10:05.106609 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.106577    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"]
Jul 20 10:10:05.174645 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.174610    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"]
Jul 20 10:10:05.178232 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.178210    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.180979 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.180949    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-d73f8-kube-rbac-proxy-sar-config\""
Jul 20 10:10:05.181180 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.181162    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-d73f8-predictor-serving-cert\""
Jul 20 10:10:05.190866 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.190840    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"]
Jul 20 10:10:05.192799 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.192769    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-cabundle-cert\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.192925 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.192829    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3166688-b723-458a-818f-b367cbeb37fa-proxy-tls\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.192925 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.192870    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.192925 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.192892    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3166688-b723-458a-818f-b367cbeb37fa-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.193079 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.192977    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqbk\" (UniqueName: \"kubernetes.io/projected/e3166688-b723-458a-818f-b367cbeb37fa-kube-api-access-cwqbk\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.294202 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294113    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-cabundle-cert\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.294202 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294159    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.294202 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294191    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3166688-b723-458a-818f-b367cbeb37fa-proxy-tls\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.294501 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294214    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.294501 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294233    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4g9\" (UniqueName: \"kubernetes.io/projected/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-api-access-4l4g9\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.294501 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294263    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3166688-b723-458a-818f-b367cbeb37fa-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.294501 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294284    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqbk\" (UniqueName: \"kubernetes.io/projected/e3166688-b723-458a-818f-b367cbeb37fa-kube-api-access-cwqbk\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.294501 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294302    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.294773 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294509    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.294773 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294570    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-proxy-tls\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.294773 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294688    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3166688-b723-458a-818f-b367cbeb37fa-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.294924 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294896    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-cabundle-cert\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.294979 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.294921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.296995 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.296972    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3166688-b723-458a-818f-b367cbeb37fa-proxy-tls\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.305059 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.305039    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqbk\" (UniqueName: \"kubernetes.io/projected/e3166688-b723-458a-818f-b367cbeb37fa-kube-api-access-cwqbk\") pod \"isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.395262 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.395211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.395442 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.395309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4g9\" (UniqueName: \"kubernetes.io/projected/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-api-access-4l4g9\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.395442 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.395349    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.395442 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.395406    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.395565 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.395438    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-proxy-tls\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.395613 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:10:05.395598    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-serving-cert: secret "isvc-xgboost-graph-raw-d73f8-predictor-serving-cert" not found
Jul 20 10:10:05.395705 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:10:05.395692    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-proxy-tls podName:e734a3f6-4bdc-40f7-a417-8a5a72f7907f nodeName:}" failed. No retries permitted until 2026-07-20 10:10:05.895636403 +0000 UTC m=+778.946565739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-proxy-tls") pod "isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" (UID: "e734a3f6-4bdc-40f7-a417-8a5a72f7907f") : secret "isvc-xgboost-graph-raw-d73f8-predictor-serving-cert" not found
Jul 20 10:10:05.395705 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.395689    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.396016 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.395996    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.396052 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.396042    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.404757 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.404733    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:05.408582 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.408560    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4g9\" (UniqueName: \"kubernetes.io/projected/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-api-access-4l4g9\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.557930 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.557840    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"]
Jul 20 10:10:05.560433 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:10:05.560402    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3166688_b723_458a_818f_b367cbeb37fa.slice/crio-10788c8efd31c7737d6a12d99fcfea7e5ee1cc70a54542bbc4aa629c16c7fd7b WatchSource:0}: Error finding container 10788c8efd31c7737d6a12d99fcfea7e5ee1cc70a54542bbc4aa629c16c7fd7b: Status 404 returned error can't find the container with id 10788c8efd31c7737d6a12d99fcfea7e5ee1cc70a54542bbc4aa629c16c7fd7b
Jul 20 10:10:05.900854 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.900815    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-proxy-tls\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.903183 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.903154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-proxy-tls\") pod \"isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:05.955034 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.954997    2555 generic.go:356] "Generic (PLEG): container finished" podID="32dba575-515b-4185-bb26-4e4a59694f1d" containerID="b37831f482e7b1b9a6544936cc3f2f3f95f0c3836e4cab3da424c163ccf9de6a" exitCode=2
Jul 20 10:10:05.955240 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.955066    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" event={"ID":"32dba575-515b-4185-bb26-4e4a59694f1d","Type":"ContainerDied","Data":"b37831f482e7b1b9a6544936cc3f2f3f95f0c3836e4cab3da424c163ccf9de6a"}
Jul 20 10:10:05.956594 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.956564    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" event={"ID":"e3166688-b723-458a-818f-b367cbeb37fa","Type":"ContainerStarted","Data":"fbad36731f24f0ff09d2c78bca949191f63a5ee9b3115a3d1517105a3e9e4e1a"}
Jul 20 10:10:05.956725 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:05.956614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" event={"ID":"e3166688-b723-458a-818f-b367cbeb37fa","Type":"ContainerStarted","Data":"10788c8efd31c7737d6a12d99fcfea7e5ee1cc70a54542bbc4aa629c16c7fd7b"}
Jul 20 10:10:06.089205 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:06.089160    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:06.227605 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:06.227523    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"]
Jul 20 10:10:06.229872 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:10:06.229837    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode734a3f6_4bdc_40f7_a417_8a5a72f7907f.slice/crio-64b6e3671977aff64c965a04ffb157df90709fadf30b155d3c807dea66a6d739 WatchSource:0}: Error finding container 64b6e3671977aff64c965a04ffb157df90709fadf30b155d3c807dea66a6d739: Status 404 returned error can't find the container with id 64b6e3671977aff64c965a04ffb157df90709fadf30b155d3c807dea66a6d739
Jul 20 10:10:06.903267 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:06.903206    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:06.963135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:06.963096    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" event={"ID":"e734a3f6-4bdc-40f7-a417-8a5a72f7907f","Type":"ContainerStarted","Data":"f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a"}
Jul 20 10:10:06.963315 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:06.963164    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" event={"ID":"e734a3f6-4bdc-40f7-a417-8a5a72f7907f","Type":"ContainerStarted","Data":"64b6e3671977aff64c965a04ffb157df90709fadf30b155d3c807dea66a6d739"}
Jul 20 10:10:08.549882 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:08.549838    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 20 10:10:09.977877 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:09.977841    2555 generic.go:356] "Generic (PLEG): container finished" podID="e3166688-b723-458a-818f-b367cbeb37fa" containerID="fbad36731f24f0ff09d2c78bca949191f63a5ee9b3115a3d1517105a3e9e4e1a" exitCode=0
Jul 20 10:10:09.977877 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:09.977866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" event={"ID":"e3166688-b723-458a-818f-b367cbeb37fa","Type":"ContainerDied","Data":"fbad36731f24f0ff09d2c78bca949191f63a5ee9b3115a3d1517105a3e9e4e1a"}
Jul 20 10:10:10.985551 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:10.985517    2555 generic.go:356] "Generic (PLEG): container finished" podID="32dba575-515b-4185-bb26-4e4a59694f1d" containerID="d1c7de8457aeba48aeb224bf38f5a1565219dc0f3e06ba9da556c3cbdd57e1c8" exitCode=0
Jul 20 10:10:10.985963 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:10.985586    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" event={"ID":"32dba575-515b-4185-bb26-4e4a59694f1d","Type":"ContainerDied","Data":"d1c7de8457aeba48aeb224bf38f5a1565219dc0f3e06ba9da556c3cbdd57e1c8"}
Jul 20 10:10:10.987056 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:10.987032    2555 generic.go:356] "Generic (PLEG): container finished" podID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerID="f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a" exitCode=0
Jul 20 10:10:10.987152 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:10.987106    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" event={"ID":"e734a3f6-4bdc-40f7-a417-8a5a72f7907f","Type":"ContainerDied","Data":"f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a"}
Jul 20 10:10:10.989487 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:10.989465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" event={"ID":"e3166688-b723-458a-818f-b367cbeb37fa","Type":"ContainerStarted","Data":"b3591a10604a9ef6b2de9b8326baa8b1ab2a43bf2edcb66b7b91ed274b24c25b"}
Jul 20 10:10:10.989563 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:10.989517    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" event={"ID":"e3166688-b723-458a-818f-b367cbeb37fa","Type":"ContainerStarted","Data":"6721deeb39204f361676c369ebf8c3ff2d31c93d71e0e92d128db6e1ec95e0d1"}
Jul 20 10:10:10.989768 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:10.989751    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:11.027153 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:11.027099    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podStartSLOduration=2.3735214989999998 podStartE2EDuration="6.027080737s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.653559238s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:10:11.026507088 +0000 UTC m=+784.077436443" watchObservedRunningTime="2026-07-20 10:10:11.027080737 +0000 UTC m=+784.078010089"
Jul 20 10:10:11.995359 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:11.995286    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:11.996757 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:11.996702    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 20 10:10:13.004567 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:13.004516    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 20 10:10:13.549690 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:13.549643    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 20 10:10:13.555575 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:13.555530    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:10:13.555783 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:13.555649    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:10:18.009618 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:18.009585    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:10:18.010401 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:18.010368    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 20 10:10:18.550015 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:18.549965    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 20 10:10:18.550213 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:18.550105    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:10:23.549935 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:23.549896    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 20 10:10:23.555651 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:23.555611    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:10:23.555824 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:23.555690    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:10:28.011185 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:28.011138    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 20 10:10:28.549732 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:28.549678    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 20 10:10:29.094105 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:29.094012    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" event={"ID":"e734a3f6-4bdc-40f7-a417-8a5a72f7907f","Type":"ContainerStarted","Data":"03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79"}
Jul 20 10:10:29.094105 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:29.094088    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" event={"ID":"e734a3f6-4bdc-40f7-a417-8a5a72f7907f","Type":"ContainerStarted","Data":"c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2"}
Jul 20 10:10:29.094660 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:29.094400    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:29.116796 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:29.116741    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podStartSLOduration=2.556017147 podStartE2EDuration="24.11672509s" totalImagesPullingTime="17.796556546s" totalInitContainerRuntime="3.764151397s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:10:29.114380094 +0000 UTC m=+802.165309474" watchObservedRunningTime="2026-07-20 10:10:29.11672509 +0000 UTC m=+802.167654444"
Jul 20 10:10:30.099754 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:30.099706    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:30.101061 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:30.101036    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 20 10:10:31.104425 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:31.104388    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 20 10:10:33.549740 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:33.549692    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 20 10:10:33.555593 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:33.555559    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 20 10:10:33.555726 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:33.555586    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:10:33.555726 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:33.555673    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:10:33.555881 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:33.555759    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:10:35.126106 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.126071    2555 generic.go:356] "Generic (PLEG): container finished" podID="32dba575-515b-4185-bb26-4e4a59694f1d" containerID="132394e79261ad7f9e89fd36debd39cc16572c2e2abc6472d774512cc6861cbc" exitCode=0
Jul 20 10:10:35.126622 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.126140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" event={"ID":"32dba575-515b-4185-bb26-4e4a59694f1d","Type":"ContainerDied","Data":"132394e79261ad7f9e89fd36debd39cc16572c2e2abc6472d774512cc6861cbc"}
Jul 20 10:10:35.677286 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.677229    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:10:35.769778 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.769691    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-cabundle-cert\") pod \"32dba575-515b-4185-bb26-4e4a59694f1d\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") "
Jul 20 10:10:35.769778 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.769734    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-kube-rbac-proxy-sar-config\") pod \"32dba575-515b-4185-bb26-4e4a59694f1d\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") "
Jul 20 10:10:35.770012 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.769786    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32dba575-515b-4185-bb26-4e4a59694f1d-kserve-provision-location\") pod \"32dba575-515b-4185-bb26-4e4a59694f1d\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") "
Jul 20 10:10:35.770012 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.769818    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32dba575-515b-4185-bb26-4e4a59694f1d-proxy-tls\") pod \"32dba575-515b-4185-bb26-4e4a59694f1d\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") "
Jul 20 10:10:35.770012 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.769844    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzpz2\" (UniqueName: \"kubernetes.io/projected/32dba575-515b-4185-bb26-4e4a59694f1d-kube-api-access-xzpz2\") pod \"32dba575-515b-4185-bb26-4e4a59694f1d\" (UID: \"32dba575-515b-4185-bb26-4e4a59694f1d\") "
Jul 20 10:10:35.770174 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.770141    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dba575-515b-4185-bb26-4e4a59694f1d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32dba575-515b-4185-bb26-4e4a59694f1d" (UID: "32dba575-515b-4185-bb26-4e4a59694f1d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:10:35.770174 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.770140    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32dba575-515b-4185-bb26-4e4a59694f1d" (UID: "32dba575-515b-4185-bb26-4e4a59694f1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:10:35.770295 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.770164    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32dba575-515b-4185-bb26-4e4a59694f1d" (UID: "32dba575-515b-4185-bb26-4e4a59694f1d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:10:35.772104 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.772073    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32dba575-515b-4185-bb26-4e4a59694f1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32dba575-515b-4185-bb26-4e4a59694f1d" (UID: "32dba575-515b-4185-bb26-4e4a59694f1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:10:35.772223 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.772181    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32dba575-515b-4185-bb26-4e4a59694f1d-kube-api-access-xzpz2" (OuterVolumeSpecName: "kube-api-access-xzpz2") pod "32dba575-515b-4185-bb26-4e4a59694f1d" (UID: "32dba575-515b-4185-bb26-4e4a59694f1d"). InnerVolumeSpecName "kube-api-access-xzpz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:10:35.871200 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.871158    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:10:35.871200 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.871193    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32dba575-515b-4185-bb26-4e4a59694f1d-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:10:35.871200 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.871204    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32dba575-515b-4185-bb26-4e4a59694f1d-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:10:35.871200 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.871216    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzpz2\" (UniqueName: \"kubernetes.io/projected/32dba575-515b-4185-bb26-4e4a59694f1d-kube-api-access-xzpz2\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:10:35.871495 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:35.871224    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32dba575-515b-4185-bb26-4e4a59694f1d-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:10:36.109498 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:36.109464    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:10:36.110168 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:36.110138    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 20 10:10:36.134552 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:36.134517    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p" event={"ID":"32dba575-515b-4185-bb26-4e4a59694f1d","Type":"ContainerDied","Data":"24b3bd87c7a676e323dbc5edd3b26296193f3ceb973cef95e68b0cfcb028f6ee"}
Jul 20 10:10:36.135003 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:36.134604    2555 scope.go:117] "RemoveContainer" containerID="132394e79261ad7f9e89fd36debd39cc16572c2e2abc6472d774512cc6861cbc"
Jul 20 10:10:36.135003 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:36.134526    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"
Jul 20 10:10:36.144514 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:36.144486    2555 scope.go:117] "RemoveContainer" containerID="b37831f482e7b1b9a6544936cc3f2f3f95f0c3836e4cab3da424c163ccf9de6a"
Jul 20 10:10:36.152942 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:36.152923    2555 scope.go:117] "RemoveContainer" containerID="d1c7de8457aeba48aeb224bf38f5a1565219dc0f3e06ba9da556c3cbdd57e1c8"
Jul 20 10:10:36.160740 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:36.160713    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"]
Jul 20 10:10:36.161489 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:36.161471    2555 scope.go:117] "RemoveContainer" containerID="86a42f804a84fc87405021da39f92a03d9fc5edd40021f78a471d5ce21de692b"
Jul 20 10:10:36.165640 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:36.165610    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a9709-predictor-d685dcf9b-8c74p"]
Jul 20 10:10:37.491886 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:37.491858    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" path="/var/lib/kubelet/pods/32dba575-515b-4185-bb26-4e4a59694f1d/volumes"
Jul 20 10:10:38.011087 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:38.011044    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 20 10:10:46.110846 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:46.110803    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 20 10:10:48.010797 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:48.010754    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 20 10:10:56.111017 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:56.110972    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 20 10:10:58.010898 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:10:58.010861    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 20 10:11:06.110300 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:06.110231    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 20 10:11:08.010708 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:08.010657    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 20 10:11:16.110466 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:16.110420    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 20 10:11:18.011236 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:18.011189    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 20 10:11:26.110383 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:26.110339    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 20 10:11:28.010894 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:28.010854    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 20 10:11:33.491333 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:33.491303    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:11:36.111419 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:36.111382    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:11:45.145049 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:45.145010    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:45.370530 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:45.370493    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:55.199838 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.199802    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:55.372067 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.372028    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:55.388839 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.388805    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"]
Jul 20 10:11:55.389081 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.389039    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container" containerID="cri-o://6721deeb39204f361676c369ebf8c3ff2d31c93d71e0e92d128db6e1ec95e0d1" gracePeriod=30
Jul 20 10:11:55.389229 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.389076    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kube-rbac-proxy" containerID="cri-o://b3591a10604a9ef6b2de9b8326baa8b1ab2a43bf2edcb66b7b91ed274b24c25b" gracePeriod=30
Jul 20 10:11:55.440810 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.440780    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:55.472826 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.472790    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"]
Jul 20 10:11:55.473376 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473359    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="storage-initializer"
Jul 20 10:11:55.473425 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473380    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="storage-initializer"
Jul 20 10:11:55.473425 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473400    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container"
Jul 20 10:11:55.473425 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473409    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container"
Jul 20 10:11:55.473520 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473430    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kube-rbac-proxy"
Jul 20 10:11:55.473520 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473439    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kube-rbac-proxy"
Jul 20 10:11:55.473520 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473458    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent"
Jul 20 10:11:55.473520 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473467    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent"
Jul 20 10:11:55.473636 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473550    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="agent"
Jul 20 10:11:55.473636 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473562    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kserve-container"
Jul 20 10:11:55.473636 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.473576    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="32dba575-515b-4185-bb26-4e4a59694f1d" containerName="kube-rbac-proxy"
Jul 20 10:11:55.477561 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.477535    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.484381 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.484354    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-serving-cert\""
Jul 20 10:11:55.485400 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.485376    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-0bd3d-kube-rbac-proxy-sar-config\""
Jul 20 10:11:55.493022 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.492998    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"]
Jul 20 10:11:55.504667 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.504634    2555 generic.go:356] "Generic (PLEG): container finished" podID="e3166688-b723-458a-818f-b367cbeb37fa" containerID="b3591a10604a9ef6b2de9b8326baa8b1ab2a43bf2edcb66b7b91ed274b24c25b" exitCode=2
Jul 20 10:11:55.504800 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.504710    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" event={"ID":"e3166688-b723-458a-818f-b367cbeb37fa","Type":"ContainerDied","Data":"b3591a10604a9ef6b2de9b8326baa8b1ab2a43bf2edcb66b7b91ed274b24c25b"}
Jul 20 10:11:55.516998 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.516971    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:55.596018 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.595976    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.596018 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.596021    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.596281 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.596161    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60b15f70-56fa-479a-a8a8-8ae06c26890a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.596281 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.596210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b15f70-56fa-479a-a8a8-8ae06c26890a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.596404 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.596358    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k429b\" (UniqueName: \"kubernetes.io/projected/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-api-access-k429b\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.600557 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.600529    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:55.697603 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.697567    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.697780 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.697614    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.697780 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.697705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60b15f70-56fa-479a-a8a8-8ae06c26890a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.697780 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.697734    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b15f70-56fa-479a-a8a8-8ae06c26890a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.697985 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.697779    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k429b\" (UniqueName: \"kubernetes.io/projected/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-api-access-k429b\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.698139 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.698114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b15f70-56fa-479a-a8a8-8ae06c26890a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.698347 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.698328    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.698386 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.698360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.700364 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.700343    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60b15f70-56fa-479a-a8a8-8ae06c26890a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.701454 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.701428    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:55.706204 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.706182    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k429b\" (UniqueName: \"kubernetes.io/projected/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-api-access-k429b\") pod \"isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.710714 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.710683    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"]
Jul 20 10:11:55.710974 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.710950    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container" containerID="cri-o://c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2" gracePeriod=30
Jul 20 10:11:55.711056 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.711027    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kube-rbac-proxy" containerID="cri-o://03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79" gracePeriod=30
Jul 20 10:11:55.788487 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.788449    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:11:55.941135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.941108    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"]
Jul 20 10:11:55.943865 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:11:55.943831    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b15f70_56fa_479a_a8a8_8ae06c26890a.slice/crio-dc314947b22529189cf72e1b4b93fb1e993837c1369d7812613e104a1ba1f43c WatchSource:0}: Error finding container dc314947b22529189cf72e1b4b93fb1e993837c1369d7812613e104a1ba1f43c: Status 404 returned error can't find the container with id dc314947b22529189cf72e1b4b93fb1e993837c1369d7812613e104a1ba1f43c
Jul 20 10:11:55.962853 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:55.962825    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:56.104855 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:56.104811    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 20 10:11:56.110459 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:56.110424    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 20 10:11:56.511468 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:56.511425    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" event={"ID":"60b15f70-56fa-479a-a8a8-8ae06c26890a","Type":"ContainerStarted","Data":"8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2"}
Jul 20 10:11:56.511929 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:56.511507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" event={"ID":"60b15f70-56fa-479a-a8a8-8ae06c26890a","Type":"ContainerStarted","Data":"dc314947b22529189cf72e1b4b93fb1e993837c1369d7812613e104a1ba1f43c"}
Jul 20 10:11:56.513700 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:56.513675    2555 generic.go:356] "Generic (PLEG): container finished" podID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerID="03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79" exitCode=2
Jul 20 10:11:56.513824 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:56.513740    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" event={"ID":"e734a3f6-4bdc-40f7-a417-8a5a72f7907f","Type":"ContainerDied","Data":"03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79"}
Jul 20 10:11:58.005552 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:58.005500    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 20 10:11:59.684164 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:11:59.684125    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode734a3f6_4bdc_40f7_a417_8a5a72f7907f.slice/crio-c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2.scope\": RecentStats: unable to find data in memory cache]"
Jul 20 10:11:59.864446 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.864417    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:11:59.934775 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.934690    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-rbac-proxy-sar-config\") pod \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") "
Jul 20 10:11:59.934775 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.934724    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-proxy-tls\") pod \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") "
Jul 20 10:11:59.935014 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.934776    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kserve-provision-location\") pod \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") "
Jul 20 10:11:59.935014 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.934798    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l4g9\" (UniqueName: \"kubernetes.io/projected/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-api-access-4l4g9\") pod \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") "
Jul 20 10:11:59.935014 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.934850    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-cabundle-cert\") pod \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\" (UID: \"e734a3f6-4bdc-40f7-a417-8a5a72f7907f\") "
Jul 20 10:11:59.935214 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.935184    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e734a3f6-4bdc-40f7-a417-8a5a72f7907f" (UID: "e734a3f6-4bdc-40f7-a417-8a5a72f7907f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:11:59.935351 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.935184    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e734a3f6-4bdc-40f7-a417-8a5a72f7907f" (UID: "e734a3f6-4bdc-40f7-a417-8a5a72f7907f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:11:59.935351 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.935238    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e734a3f6-4bdc-40f7-a417-8a5a72f7907f" (UID: "e734a3f6-4bdc-40f7-a417-8a5a72f7907f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:11:59.936915 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.936895    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e734a3f6-4bdc-40f7-a417-8a5a72f7907f" (UID: "e734a3f6-4bdc-40f7-a417-8a5a72f7907f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:11:59.937018 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:11:59.936999    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-api-access-4l4g9" (OuterVolumeSpecName: "kube-api-access-4l4g9") pod "e734a3f6-4bdc-40f7-a417-8a5a72f7907f" (UID: "e734a3f6-4bdc-40f7-a417-8a5a72f7907f"). InnerVolumeSpecName "kube-api-access-4l4g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:12:00.036052 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.036008    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:12:00.036052 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.036046    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:12:00.036052 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.036061    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:12:00.036336 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.036073    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l4g9\" (UniqueName: \"kubernetes.io/projected/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-kube-api-access-4l4g9\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:12:00.036336 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.036084    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e734a3f6-4bdc-40f7-a417-8a5a72f7907f-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:12:00.532875 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.532838    2555 generic.go:356] "Generic (PLEG): container finished" podID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerID="8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2" exitCode=0
Jul 20 10:12:00.533106 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.532915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" event={"ID":"60b15f70-56fa-479a-a8a8-8ae06c26890a","Type":"ContainerDied","Data":"8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2"}
Jul 20 10:12:00.535343 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.535289    2555 generic.go:356] "Generic (PLEG): container finished" podID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerID="c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2" exitCode=0
Jul 20 10:12:00.535343 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.535339    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"
Jul 20 10:12:00.535535 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.535407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" event={"ID":"e734a3f6-4bdc-40f7-a417-8a5a72f7907f","Type":"ContainerDied","Data":"c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2"}
Jul 20 10:12:00.535535 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.535474    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb" event={"ID":"e734a3f6-4bdc-40f7-a417-8a5a72f7907f","Type":"ContainerDied","Data":"64b6e3671977aff64c965a04ffb157df90709fadf30b155d3c807dea66a6d739"}
Jul 20 10:12:00.535535 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.535508    2555 scope.go:117] "RemoveContainer" containerID="03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79"
Jul 20 10:12:00.538234 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.538035    2555 generic.go:356] "Generic (PLEG): container finished" podID="e3166688-b723-458a-818f-b367cbeb37fa" containerID="6721deeb39204f361676c369ebf8c3ff2d31c93d71e0e92d128db6e1ec95e0d1" exitCode=0
Jul 20 10:12:00.538234 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.538100    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" event={"ID":"e3166688-b723-458a-818f-b367cbeb37fa","Type":"ContainerDied","Data":"6721deeb39204f361676c369ebf8c3ff2d31c93d71e0e92d128db6e1ec95e0d1"}
Jul 20 10:12:00.545208 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.545191    2555 scope.go:117] "RemoveContainer" containerID="c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2"
Jul 20 10:12:00.555031 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.555010    2555 scope.go:117] "RemoveContainer" containerID="f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a"
Jul 20 10:12:00.564911 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.564863    2555 scope.go:117] "RemoveContainer" containerID="03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79"
Jul 20 10:12:00.565533 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:12:00.565425    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79\": container with ID starting with 03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79 not found: ID does not exist" containerID="03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79"
Jul 20 10:12:00.565533 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.565465    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79"} err="failed to get container status \"03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79\": rpc error: code = NotFound desc = could not find container \"03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79\": container with ID starting with 03213aa21a7c8aca14313e3aad83865e6e3c122b8c9aee17d30906a83b630e79 not found: ID does not exist"
Jul 20 10:12:00.565533 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.565494    2555 scope.go:117] "RemoveContainer" containerID="c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2"
Jul 20 10:12:00.565917 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:12:00.565844    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2\": container with ID starting with c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2 not found: ID does not exist" containerID="c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2"
Jul 20 10:12:00.565917 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.565870    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2"} err="failed to get container status \"c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2\": rpc error: code = NotFound desc = could not find container \"c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2\": container with ID starting with c075c066bd456b8e58c2659460ea8f2ec4f07a203b8cb768a0d01155867072f2 not found: ID does not exist"
Jul 20 10:12:00.565917 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.565912    2555 scope.go:117] "RemoveContainer" containerID="f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a"
Jul 20 10:12:00.566517 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:12:00.566311    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a\": container with ID starting with f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a not found: ID does not exist" containerID="f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a"
Jul 20 10:12:00.566517 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.566342    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a"} err="failed to get container status \"f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a\": rpc error: code = NotFound desc = could not find container \"f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a\": container with ID starting with f12d1efbce3539cb5853cf0b8d785e0022549db0324f1e9914fce053166b174a not found: ID does not exist"
Jul 20 10:12:00.566853 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.566829    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"]
Jul 20 10:12:00.574828 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.574794    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-d73f8-predictor-5d6dc79f9c-88kjb"]
Jul 20 10:12:00.624197 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.624172    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:12:00.640932 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.640906    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-kube-rbac-proxy-sar-config\") pod \"e3166688-b723-458a-818f-b367cbeb37fa\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") "
Jul 20 10:12:00.641053 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.640954    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-cabundle-cert\") pod \"e3166688-b723-458a-818f-b367cbeb37fa\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") "
Jul 20 10:12:00.641053 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.640997    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwqbk\" (UniqueName: \"kubernetes.io/projected/e3166688-b723-458a-818f-b367cbeb37fa-kube-api-access-cwqbk\") pod \"e3166688-b723-458a-818f-b367cbeb37fa\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") "
Jul 20 10:12:00.641053 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.641048    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3166688-b723-458a-818f-b367cbeb37fa-proxy-tls\") pod \"e3166688-b723-458a-818f-b367cbeb37fa\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") "
Jul 20 10:12:00.641186 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.641094    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3166688-b723-458a-818f-b367cbeb37fa-kserve-provision-location\") pod \"e3166688-b723-458a-818f-b367cbeb37fa\" (UID: \"e3166688-b723-458a-818f-b367cbeb37fa\") "
Jul 20 10:12:00.641343 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.641315    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3166688-b723-458a-818f-b367cbeb37fa" (UID: "e3166688-b723-458a-818f-b367cbeb37fa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:12:00.641429 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.641355    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3166688-b723-458a-818f-b367cbeb37fa" (UID: "e3166688-b723-458a-818f-b367cbeb37fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:12:00.641517 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.641496    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3166688-b723-458a-818f-b367cbeb37fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3166688-b723-458a-818f-b367cbeb37fa" (UID: "e3166688-b723-458a-818f-b367cbeb37fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:12:00.643656 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.643627    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3166688-b723-458a-818f-b367cbeb37fa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3166688-b723-458a-818f-b367cbeb37fa" (UID: "e3166688-b723-458a-818f-b367cbeb37fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:12:00.643759 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.643721    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3166688-b723-458a-818f-b367cbeb37fa-kube-api-access-cwqbk" (OuterVolumeSpecName: "kube-api-access-cwqbk") pod "e3166688-b723-458a-818f-b367cbeb37fa" (UID: "e3166688-b723-458a-818f-b367cbeb37fa"). InnerVolumeSpecName "kube-api-access-cwqbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:12:00.742485 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.742446    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3166688-b723-458a-818f-b367cbeb37fa-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:12:00.742485 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.742484    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:12:00.743002 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.742497    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3166688-b723-458a-818f-b367cbeb37fa-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:12:00.743002 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.742510    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwqbk\" (UniqueName: \"kubernetes.io/projected/e3166688-b723-458a-818f-b367cbeb37fa-kube-api-access-cwqbk\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:12:00.743002 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:00.742520    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3166688-b723-458a-818f-b367cbeb37fa-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:12:01.492136 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.492103    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" path="/var/lib/kubelet/pods/e734a3f6-4bdc-40f7-a417-8a5a72f7907f/volumes"
Jul 20 10:12:01.547096 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.547059    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"
Jul 20 10:12:01.547269 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.547063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88" event={"ID":"e3166688-b723-458a-818f-b367cbeb37fa","Type":"ContainerDied","Data":"10788c8efd31c7737d6a12d99fcfea7e5ee1cc70a54542bbc4aa629c16c7fd7b"}
Jul 20 10:12:01.547269 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.547232    2555 scope.go:117] "RemoveContainer" containerID="b3591a10604a9ef6b2de9b8326baa8b1ab2a43bf2edcb66b7b91ed274b24c25b"
Jul 20 10:12:01.550108 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.550051    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" event={"ID":"60b15f70-56fa-479a-a8a8-8ae06c26890a","Type":"ContainerStarted","Data":"9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b"}
Jul 20 10:12:01.550261 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.550144    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" event={"ID":"60b15f70-56fa-479a-a8a8-8ae06c26890a","Type":"ContainerStarted","Data":"b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500"}
Jul 20 10:12:01.550637 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.550618    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:12:01.550890 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.550846    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:12:01.552088 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.552060    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 20 10:12:01.556313 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.556276    2555 scope.go:117] "RemoveContainer" containerID="6721deeb39204f361676c369ebf8c3ff2d31c93d71e0e92d128db6e1ec95e0d1"
Jul 20 10:12:01.564028 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.564008    2555 scope.go:117] "RemoveContainer" containerID="fbad36731f24f0ff09d2c78bca949191f63a5ee9b3115a3d1517105a3e9e4e1a"
Jul 20 10:12:01.580761 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.580727    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"]
Jul 20 10:12:01.590522 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:01.590492    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-d73f8-predictor-76dbbfdc88-8pd88"]
Jul 20 10:12:02.557229 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:02.557189    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 20 10:12:03.491652 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:03.491620    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3166688-b723-458a-818f-b367cbeb37fa" path="/var/lib/kubelet/pods/e3166688-b723-458a-818f-b367cbeb37fa/volumes"
Jul 20 10:12:07.458136 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:07.458100    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:12:07.458675 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:07.458291    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:12:07.561518 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:07.561487    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:12:07.562024 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:07.561999    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 20 10:12:07.587047 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:07.586993    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podStartSLOduration=8.902179728 podStartE2EDuration="12.586979266s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684799538s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:12:01.621136822 +0000 UTC m=+894.672066177" watchObservedRunningTime="2026-07-20 10:12:07.586979266 +0000 UTC m=+900.637908622"
Jul 20 10:12:17.562512 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:17.562470    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 20 10:12:22.365032 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:22.364996    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:27.562363 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:27.562309    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 20 10:12:37.562044 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:37.561998    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 20 10:12:47.562393 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:47.562350    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 20 10:12:57.562060 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:12:57.562013    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 20 10:13:07.562169 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:07.562130    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 20 10:13:12.486993 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:12.486952    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 20 10:13:22.488600 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:22.488567    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:13:35.480439 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:35.480398    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:35.884271 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:35.884218    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:45.639232 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:45.639191    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:45.975106 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:45.975015    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:46.081700 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.081664    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"]
Jul 20 10:13:46.082062 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082048    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="storage-initializer"
Jul 20 10:13:46.082135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082066    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="storage-initializer"
Jul 20 10:13:46.082135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082081    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kube-rbac-proxy"
Jul 20 10:13:46.082135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082091    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kube-rbac-proxy"
Jul 20 10:13:46.082135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082102    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container"
Jul 20 10:13:46.082135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082107    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container"
Jul 20 10:13:46.082135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082118    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="storage-initializer"
Jul 20 10:13:46.082135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082123    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="storage-initializer"
Jul 20 10:13:46.082135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082131    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kube-rbac-proxy"
Jul 20 10:13:46.082135 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082139    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kube-rbac-proxy"
Jul 20 10:13:46.082440 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082152    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container"
Jul 20 10:13:46.082440 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082159    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container"
Jul 20 10:13:46.082440 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082222    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kserve-container"
Jul 20 10:13:46.082440 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082233    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e734a3f6-4bdc-40f7-a417-8a5a72f7907f" containerName="kube-rbac-proxy"
Jul 20 10:13:46.082440 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082241    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kube-rbac-proxy"
Jul 20 10:13:46.082440 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.082263    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3166688-b723-458a-818f-b367cbeb37fa" containerName="kserve-container"
Jul 20 10:13:46.085548 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.085530    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.092785 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.092764    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-22c6a-kube-rbac-proxy-sar-config\""
Jul 20 10:13:46.092900 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.092800    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-22c6a-predictor-serving-cert\""
Jul 20 10:13:46.106152 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.106115    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"]
Jul 20 10:13:46.186452 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.186410    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51afa2e-5ce2-42d9-99bc-30bda0468137-proxy-tls\") pod \"message-dumper-raw-22c6a-predictor-7597674fb-vwxnt\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") " pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.186660 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.186532    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-22c6a-predictor-7597674fb-vwxnt\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") " pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.186660 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.186561    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwh28\" (UniqueName: \"kubernetes.io/projected/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-api-access-hwh28\") pod \"message-dumper-raw-22c6a-predictor-7597674fb-vwxnt\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") " pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.198912 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.198875    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"]
Jul 20 10:13:46.199204 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.199175    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container" containerID="cri-o://b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500" gracePeriod=30
Jul 20 10:13:46.199302 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.199235    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kube-rbac-proxy" containerID="cri-o://9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b" gracePeriod=30
Jul 20 10:13:46.230495 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.230399    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:46.288108 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.288072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-22c6a-predictor-7597674fb-vwxnt\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") " pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.288108 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.288110    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwh28\" (UniqueName: \"kubernetes.io/projected/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-api-access-hwh28\") pod \"message-dumper-raw-22c6a-predictor-7597674fb-vwxnt\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") " pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.288453 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.288177    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51afa2e-5ce2-42d9-99bc-30bda0468137-proxy-tls\") pod \"message-dumper-raw-22c6a-predictor-7597674fb-vwxnt\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") " pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.288838 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.288811    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-22c6a-predictor-7597674fb-vwxnt\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") " pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.290754 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.290730    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51afa2e-5ce2-42d9-99bc-30bda0468137-proxy-tls\") pod \"message-dumper-raw-22c6a-predictor-7597674fb-vwxnt\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") " pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.323864 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.323832    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwh28\" (UniqueName: \"kubernetes.io/projected/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-api-access-hwh28\") pod \"message-dumper-raw-22c6a-predictor-7597674fb-vwxnt\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") " pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.378543 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.378482    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:46.396046 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.396004    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:46.469317 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.469282    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:46.548323 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.548283    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"]
Jul 20 10:13:46.548553 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:13:46.548529    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51afa2e_5ce2_42d9_99bc_30bda0468137.slice/crio-814da4dfca0a7376a27a24a89145e0c3c52c59354218eda13809d3561311be6a WatchSource:0}: Error finding container 814da4dfca0a7376a27a24a89145e0c3c52c59354218eda13809d3561311be6a: Status 404 returned error can't find the container with id 814da4dfca0a7376a27a24a89145e0c3c52c59354218eda13809d3561311be6a
Jul 20 10:13:46.550449 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:46.550431    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:13:47.033604 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:47.033562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt" event={"ID":"a51afa2e-5ce2-42d9-99bc-30bda0468137","Type":"ContainerStarted","Data":"814da4dfca0a7376a27a24a89145e0c3c52c59354218eda13809d3561311be6a"}
Jul 20 10:13:47.035780 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:47.035754    2555 generic.go:356] "Generic (PLEG): container finished" podID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerID="9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b" exitCode=2
Jul 20 10:13:47.035927 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:47.035805    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" event={"ID":"60b15f70-56fa-479a-a8a8-8ae06c26890a","Type":"ContainerDied","Data":"9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b"}
Jul 20 10:13:47.558064 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:47.558009    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 20 10:13:48.044019 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:48.043976    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt" event={"ID":"a51afa2e-5ce2-42d9-99bc-30bda0468137","Type":"ContainerStarted","Data":"ef52a2d81a26d67d27b64fe415ef5ec010b8a4fb75cc1ec8d46e3898184668b5"}
Jul 20 10:13:48.044517 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:48.044064    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt" event={"ID":"a51afa2e-5ce2-42d9-99bc-30bda0468137","Type":"ContainerStarted","Data":"2765fb4c6cdcfcf3906f552f884458d36051724553e4e598d16d76ae67bc0bb4"}
Jul 20 10:13:48.044517 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:48.044102    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:48.068211 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:48.068157    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt" podStartSLOduration=0.788631181 podStartE2EDuration="2.068139033s" totalImagesPullingTime="1.279507852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:13:48.06773734 +0000 UTC m=+1001.118666676" watchObservedRunningTime="2026-07-20 10:13:48.068139033 +0000 UTC m=+1001.119068479"
Jul 20 10:13:49.048914 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:49.048875    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:49.050905 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:49.050883    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:13:51.456436 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.456404    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:13:51.535790 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.535691    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-rbac-proxy-sar-config\") pod \"60b15f70-56fa-479a-a8a8-8ae06c26890a\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") "
Jul 20 10:13:51.535790 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.535751    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-cabundle-cert\") pod \"60b15f70-56fa-479a-a8a8-8ae06c26890a\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") "
Jul 20 10:13:51.535790 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.535791    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b15f70-56fa-479a-a8a8-8ae06c26890a-kserve-provision-location\") pod \"60b15f70-56fa-479a-a8a8-8ae06c26890a\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") "
Jul 20 10:13:51.536079 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.535826    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60b15f70-56fa-479a-a8a8-8ae06c26890a-proxy-tls\") pod \"60b15f70-56fa-479a-a8a8-8ae06c26890a\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") "
Jul 20 10:13:51.536079 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.535876    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k429b\" (UniqueName: \"kubernetes.io/projected/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-api-access-k429b\") pod \"60b15f70-56fa-479a-a8a8-8ae06c26890a\" (UID: \"60b15f70-56fa-479a-a8a8-8ae06c26890a\") "
Jul 20 10:13:51.536186 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.536136    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "60b15f70-56fa-479a-a8a8-8ae06c26890a" (UID: "60b15f70-56fa-479a-a8a8-8ae06c26890a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:13:51.536186 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.536146    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b15f70-56fa-479a-a8a8-8ae06c26890a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60b15f70-56fa-479a-a8a8-8ae06c26890a" (UID: "60b15f70-56fa-479a-a8a8-8ae06c26890a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:51.536186 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.536143    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "60b15f70-56fa-479a-a8a8-8ae06c26890a" (UID: "60b15f70-56fa-479a-a8a8-8ae06c26890a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:13:51.538068 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.538031    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b15f70-56fa-479a-a8a8-8ae06c26890a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "60b15f70-56fa-479a-a8a8-8ae06c26890a" (UID: "60b15f70-56fa-479a-a8a8-8ae06c26890a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:13:51.538175 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.538090    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-api-access-k429b" (OuterVolumeSpecName: "kube-api-access-k429b") pod "60b15f70-56fa-479a-a8a8-8ae06c26890a" (UID: "60b15f70-56fa-479a-a8a8-8ae06c26890a"). InnerVolumeSpecName "kube-api-access-k429b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:13:51.637267 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.637207    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:13:51.637267 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.637266    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60b15f70-56fa-479a-a8a8-8ae06c26890a-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:13:51.637267 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.637279    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b15f70-56fa-479a-a8a8-8ae06c26890a-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:13:51.637519 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.637289    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60b15f70-56fa-479a-a8a8-8ae06c26890a-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:13:51.637519 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:51.637300    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k429b\" (UniqueName: \"kubernetes.io/projected/60b15f70-56fa-479a-a8a8-8ae06c26890a-kube-api-access-k429b\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:13:52.064767 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.064734    2555 generic.go:356] "Generic (PLEG): container finished" podID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerID="b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500" exitCode=0
Jul 20 10:13:52.064945 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.064781    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" event={"ID":"60b15f70-56fa-479a-a8a8-8ae06c26890a","Type":"ContainerDied","Data":"b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500"}
Jul 20 10:13:52.064945 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.064796    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"
Jul 20 10:13:52.064945 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.064830    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz" event={"ID":"60b15f70-56fa-479a-a8a8-8ae06c26890a","Type":"ContainerDied","Data":"dc314947b22529189cf72e1b4b93fb1e993837c1369d7812613e104a1ba1f43c"}
Jul 20 10:13:52.064945 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.064856    2555 scope.go:117] "RemoveContainer" containerID="9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b"
Jul 20 10:13:52.074093 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.074074    2555 scope.go:117] "RemoveContainer" containerID="b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500"
Jul 20 10:13:52.082065 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.082047    2555 scope.go:117] "RemoveContainer" containerID="8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2"
Jul 20 10:13:52.088887 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.088856    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"]
Jul 20 10:13:52.090892 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.090863    2555 scope.go:117] "RemoveContainer" containerID="9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b"
Jul 20 10:13:52.091203 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:13:52.091183    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b\": container with ID starting with 9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b not found: ID does not exist" containerID="9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b"
Jul 20 10:13:52.091313 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.091211    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b"} err="failed to get container status \"9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b\": rpc error: code = NotFound desc = could not find container \"9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b\": container with ID starting with 9abb3e912bea0070e9526ce31e58561aa11e494c1b90b55a24bc0c73ec650c3b not found: ID does not exist"
Jul 20 10:13:52.091313 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.091230    2555 scope.go:117] "RemoveContainer" containerID="b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500"
Jul 20 10:13:52.091313 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.091286    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0bd3d-predictor-64d6667848-k2pmz"]
Jul 20 10:13:52.091516 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:13:52.091497    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500\": container with ID starting with b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500 not found: ID does not exist" containerID="b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500"
Jul 20 10:13:52.091556 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.091525    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500"} err="failed to get container status \"b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500\": rpc error: code = NotFound desc = could not find container \"b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500\": container with ID starting with b02d815e478a7531bdd05b777d25eddbcb69f16ef0fc6460ef28eaa2a7f37500 not found: ID does not exist"
Jul 20 10:13:52.091556 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.091543    2555 scope.go:117] "RemoveContainer" containerID="8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2"
Jul 20 10:13:52.091813 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:13:52.091798    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2\": container with ID starting with 8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2 not found: ID does not exist" containerID="8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2"
Jul 20 10:13:52.091851 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:52.091817    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2"} err="failed to get container status \"8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2\": rpc error: code = NotFound desc = could not find container \"8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2\": container with ID starting with 8856cc2f6902d82867ccc22529362ddd9165ef8e6a4de7ad393c8546fd187aa2 not found: ID does not exist"
Jul 20 10:13:53.491402 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:53.491362    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" path="/var/lib/kubelet/pods/60b15f70-56fa-479a-a8a8-8ae06c26890a/volumes"
Jul 20 10:13:56.062644 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:13:56.062615    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:14:05.909470 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:05.909432    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:06.008761 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.008719    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"]
Jul 20 10:14:06.009184 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.009165    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container"
Jul 20 10:14:06.009303 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.009188    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container"
Jul 20 10:14:06.009303 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.009207    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="storage-initializer"
Jul 20 10:14:06.009303 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.009214    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="storage-initializer"
Jul 20 10:14:06.009303 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.009236    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kube-rbac-proxy"
Jul 20 10:14:06.009303 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.009260    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kube-rbac-proxy"
Jul 20 10:14:06.009574 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.009361    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kube-rbac-proxy"
Jul 20 10:14:06.009574 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.009378    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b15f70-56fa-479a-a8a8-8ae06c26890a" containerName="kserve-container"
Jul 20 10:14:06.012748 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.012724    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.015959 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.015937    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-22c6a-predictor-serving-cert\""
Jul 20 10:14:06.017565 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.017548    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:14:06.017886 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.017867    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:14:06.019519 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.019497    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-22c6a-kube-rbac-proxy-sar-config\""
Jul 20 10:14:06.028288 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.028263    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"]
Jul 20 10:14:06.114778 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.114745    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:06.165057 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.164964    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aef7e726-3136-4c35-9eed-567c438d3ec8-kserve-provision-location\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.165057 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.165008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.165279 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.165071    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-cabundle-cert\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.165279 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.165159    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aef7e726-3136-4c35-9eed-567c438d3ec8-proxy-tls\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.165279 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.165194    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhjs\" (UniqueName: \"kubernetes.io/projected/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-api-access-5zhjs\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.266516 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.266482    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aef7e726-3136-4c35-9eed-567c438d3ec8-proxy-tls\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.266700 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.266524    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhjs\" (UniqueName: \"kubernetes.io/projected/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-api-access-5zhjs\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.266700 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.266565    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aef7e726-3136-4c35-9eed-567c438d3ec8-kserve-provision-location\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.266700 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.266596    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.266700 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.266640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-cabundle-cert\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.267066 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.267043    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aef7e726-3136-4c35-9eed-567c438d3ec8-kserve-provision-location\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.267342 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.267322    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.267395 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.267357    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-cabundle-cert\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.268970 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.268951    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aef7e726-3136-4c35-9eed-567c438d3ec8-proxy-tls\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.283145 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.283113    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhjs\" (UniqueName: \"kubernetes.io/projected/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-api-access-5zhjs\") pod \"isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.323844 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.323804    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:06.477456 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:06.477425    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"]
Jul 20 10:14:06.479754 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:14:06.479710    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef7e726_3136_4c35_9eed_567c438d3ec8.slice/crio-16f5b46c2318334ac2867feb90053b1d21752f52800349270eb5df3f79453916 WatchSource:0}: Error finding container 16f5b46c2318334ac2867feb90053b1d21752f52800349270eb5df3f79453916: Status 404 returned error can't find the container with id 16f5b46c2318334ac2867feb90053b1d21752f52800349270eb5df3f79453916
Jul 20 10:14:07.135705 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:07.135660    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" event={"ID":"aef7e726-3136-4c35-9eed-567c438d3ec8","Type":"ContainerStarted","Data":"6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3"}
Jul 20 10:14:07.136130 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:07.135735    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" event={"ID":"aef7e726-3136-4c35-9eed-567c438d3ec8","Type":"ContainerStarted","Data":"16f5b46c2318334ac2867feb90053b1d21752f52800349270eb5df3f79453916"}
Jul 20 10:14:11.156771 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:11.156736    2555 generic.go:356] "Generic (PLEG): container finished" podID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerID="6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3" exitCode=0
Jul 20 10:14:11.157171 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:11.156807    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" event={"ID":"aef7e726-3136-4c35-9eed-567c438d3ec8","Type":"ContainerDied","Data":"6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3"}
Jul 20 10:14:12.163659 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:12.163627    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" event={"ID":"aef7e726-3136-4c35-9eed-567c438d3ec8","Type":"ContainerStarted","Data":"b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378"}
Jul 20 10:14:12.164083 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:12.163695    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" event={"ID":"aef7e726-3136-4c35-9eed-567c438d3ec8","Type":"ContainerStarted","Data":"9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f"}
Jul 20 10:14:12.164083 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:12.163722    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" event={"ID":"aef7e726-3136-4c35-9eed-567c438d3ec8","Type":"ContainerStarted","Data":"c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99"}
Jul 20 10:14:12.164083 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:12.164065    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:12.164232 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:12.164211    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:12.165763 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:12.165729    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:14:12.196536 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:12.196475    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podStartSLOduration=3.5806055150000002 podStartE2EDuration="7.196453976s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.615848461s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:14:12.195795209 +0000 UTC m=+1025.246724565" watchObservedRunningTime="2026-07-20 10:14:12.196453976 +0000 UTC m=+1025.247383332"
Jul 20 10:14:13.167808 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:13.167765    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:13.168279 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:13.167824    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:14:13.169134 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:13.169100    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:14.172357 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:14.172301    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:14:14.172805 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:14.172771    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:19.177366 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:19.177335    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:14:19.177982 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:19.177860    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:14:19.178208 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:19.178186    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:29.178577 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:29.178535    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:14:29.179035 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:29.179011    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:31.734598 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:31.734562    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:39.177848 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:39.177795    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:14:39.178279 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:39.178209    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:49.178120 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:49.178080    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:14:49.178594 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:49.178551    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:59.178671 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:59.178625    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:14:59.179186 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:14:59.179110    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:15:09.178582 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:09.178538    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:15:09.179010 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:09.178955    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:15:19.177965 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:19.177922    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:15:19.178449 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:19.178345    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:15:29.178447 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:29.178410    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:15:29.178833 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:29.178498    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:15:33.545755 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:33.545715    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:40.918094 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:40.918059    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-22c6a-predictor-7597674fb-vwxnt_a51afa2e-5ce2-42d9-99bc-30bda0468137/kserve-container/0.log"
Jul 20 10:15:41.323540 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.323443    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"]
Jul 20 10:15:41.323880 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.323846    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" containerID="cri-o://c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99" gracePeriod=30
Jul 20 10:15:41.324084 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.324038    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" containerID="cri-o://b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378" gracePeriod=30
Jul 20 10:15:41.324199 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.323943    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kube-rbac-proxy" containerID="cri-o://9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f" gracePeriod=30
Jul 20 10:15:41.344752 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.344717    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"]
Jul 20 10:15:41.348593 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.348565    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.356056 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.355397    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-07030-predictor-serving-cert\""
Jul 20 10:15:41.356190 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.356165    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-07030-kube-rbac-proxy-sar-config\""
Jul 20 10:15:41.368240 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.368206    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"]
Jul 20 10:15:41.383261 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.383215    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"]
Jul 20 10:15:41.383486 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.383464    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt" podUID="a51afa2e-5ce2-42d9-99bc-30bda0468137" containerName="kserve-container" containerID="cri-o://2765fb4c6cdcfcf3906f552f884458d36051724553e4e598d16d76ae67bc0bb4" gracePeriod=30
Jul 20 10:15:41.383608 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.383559    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt" podUID="a51afa2e-5ce2-42d9-99bc-30bda0468137" containerName="kube-rbac-proxy" containerID="cri-o://ef52a2d81a26d67d27b64fe415ef5ec010b8a4fb75cc1ec8d46e3898184668b5" gracePeriod=30
Jul 20 10:15:41.471324 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.471286    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.471324 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.471324    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfjw7\" (UniqueName: \"kubernetes.io/projected/60889480-945c-43a5-bf90-accfb84f6997-kube-api-access-mfjw7\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.471572 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.471354    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60889480-945c-43a5-bf90-accfb84f6997-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.471572 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.471385    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-cabundle-cert\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.471572 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.471431    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60889480-945c-43a5-bf90-accfb84f6997-proxy-tls\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.529977 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.529926    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:41.572617 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.571985    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.572617 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.572026    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfjw7\" (UniqueName: \"kubernetes.io/projected/60889480-945c-43a5-bf90-accfb84f6997-kube-api-access-mfjw7\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.572617 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.572065    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60889480-945c-43a5-bf90-accfb84f6997-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.572617 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.572182    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-cabundle-cert\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.572617 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.572291    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60889480-945c-43a5-bf90-accfb84f6997-proxy-tls\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.572617 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.572469    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60889480-945c-43a5-bf90-accfb84f6997-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.573095 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.572868    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.573284 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.573207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-cabundle-cert\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.576409 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.576336    2555 generic.go:356] "Generic (PLEG): container finished" podID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerID="9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f" exitCode=2
Jul 20 10:15:41.576515 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.576424    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" event={"ID":"aef7e726-3136-4c35-9eed-567c438d3ec8","Type":"ContainerDied","Data":"9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f"}
Jul 20 10:15:41.577898 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.577874    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60889480-945c-43a5-bf90-accfb84f6997-proxy-tls\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.578970 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.578948    2555 generic.go:356] "Generic (PLEG): container finished" podID="a51afa2e-5ce2-42d9-99bc-30bda0468137" containerID="ef52a2d81a26d67d27b64fe415ef5ec010b8a4fb75cc1ec8d46e3898184668b5" exitCode=2
Jul 20 10:15:41.578970 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.578971    2555 generic.go:356] "Generic (PLEG): container finished" podID="a51afa2e-5ce2-42d9-99bc-30bda0468137" containerID="2765fb4c6cdcfcf3906f552f884458d36051724553e4e598d16d76ae67bc0bb4" exitCode=2
Jul 20 10:15:41.579140 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.579014    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt" event={"ID":"a51afa2e-5ce2-42d9-99bc-30bda0468137","Type":"ContainerDied","Data":"ef52a2d81a26d67d27b64fe415ef5ec010b8a4fb75cc1ec8d46e3898184668b5"}
Jul 20 10:15:41.579140 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.579073    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt" event={"ID":"a51afa2e-5ce2-42d9-99bc-30bda0468137","Type":"ContainerDied","Data":"2765fb4c6cdcfcf3906f552f884458d36051724553e4e598d16d76ae67bc0bb4"}
Jul 20 10:15:41.581557 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.581531    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfjw7\" (UniqueName: \"kubernetes.io/projected/60889480-945c-43a5-bf90-accfb84f6997-kube-api-access-mfjw7\") pod \"isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.630912 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.630886    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:15:41.631813 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.631788    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:41.659867 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.659832    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:41.672853 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.672820    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51afa2e-5ce2-42d9-99bc-30bda0468137-proxy-tls\") pod \"a51afa2e-5ce2-42d9-99bc-30bda0468137\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") "
Jul 20 10:15:41.673013 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.672876    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwh28\" (UniqueName: \"kubernetes.io/projected/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-api-access-hwh28\") pod \"a51afa2e-5ce2-42d9-99bc-30bda0468137\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") "
Jul 20 10:15:41.673013 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.672969    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-rbac-proxy-sar-config\") pod \"a51afa2e-5ce2-42d9-99bc-30bda0468137\" (UID: \"a51afa2e-5ce2-42d9-99bc-30bda0468137\") "
Jul 20 10:15:41.673457 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.673421    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a51afa2e-5ce2-42d9-99bc-30bda0468137" (UID: "a51afa2e-5ce2-42d9-99bc-30bda0468137"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:15:41.675647 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.675616    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51afa2e-5ce2-42d9-99bc-30bda0468137-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a51afa2e-5ce2-42d9-99bc-30bda0468137" (UID: "a51afa2e-5ce2-42d9-99bc-30bda0468137"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:15:41.675748 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.675624    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-api-access-hwh28" (OuterVolumeSpecName: "kube-api-access-hwh28") pod "a51afa2e-5ce2-42d9-99bc-30bda0468137" (UID: "a51afa2e-5ce2-42d9-99bc-30bda0468137"). InnerVolumeSpecName "kube-api-access-hwh28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:15:41.773741 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.773709    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:15:41.773741 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.773738    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51afa2e-5ce2-42d9-99bc-30bda0468137-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:15:41.773933 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.773753    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwh28\" (UniqueName: \"kubernetes.io/projected/a51afa2e-5ce2-42d9-99bc-30bda0468137-kube-api-access-hwh28\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:15:41.804984 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:41.804954    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"]
Jul 20 10:15:41.807404 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:15:41.807370    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60889480_945c_43a5_bf90_accfb84f6997.slice/crio-2cbbf8862fcca6f66cff85bfc8044fea6bd664c15f4a827a2281c517270c57ec WatchSource:0}: Error finding container 2cbbf8862fcca6f66cff85bfc8044fea6bd664c15f4a827a2281c517270c57ec: Status 404 returned error can't find the container with id 2cbbf8862fcca6f66cff85bfc8044fea6bd664c15f4a827a2281c517270c57ec
Jul 20 10:15:42.586261 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:42.586218    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"
Jul 20 10:15:42.586261 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:42.586219    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt" event={"ID":"a51afa2e-5ce2-42d9-99bc-30bda0468137","Type":"ContainerDied","Data":"814da4dfca0a7376a27a24a89145e0c3c52c59354218eda13809d3561311be6a"}
Jul 20 10:15:42.586767 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:42.586330    2555 scope.go:117] "RemoveContainer" containerID="ef52a2d81a26d67d27b64fe415ef5ec010b8a4fb75cc1ec8d46e3898184668b5"
Jul 20 10:15:42.587988 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:42.587957    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" event={"ID":"60889480-945c-43a5-bf90-accfb84f6997","Type":"ContainerStarted","Data":"c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293"}
Jul 20 10:15:42.588107 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:42.588024    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" event={"ID":"60889480-945c-43a5-bf90-accfb84f6997","Type":"ContainerStarted","Data":"2cbbf8862fcca6f66cff85bfc8044fea6bd664c15f4a827a2281c517270c57ec"}
Jul 20 10:15:42.595207 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:42.595191    2555 scope.go:117] "RemoveContainer" containerID="2765fb4c6cdcfcf3906f552f884458d36051724553e4e598d16d76ae67bc0bb4"
Jul 20 10:15:42.630727 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:42.630671    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"]
Jul 20 10:15:42.633994 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:42.633964    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22c6a-predictor-7597674fb-vwxnt"]
Jul 20 10:15:43.491983 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:43.491948    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51afa2e-5ce2-42d9-99bc-30bda0468137" path="/var/lib/kubelet/pods/a51afa2e-5ce2-42d9-99bc-30bda0468137/volumes"
Jul 20 10:15:44.172661 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:44.172618    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 20 10:15:46.610261 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:46.610215    2555 generic.go:356] "Generic (PLEG): container finished" podID="60889480-945c-43a5-bf90-accfb84f6997" containerID="c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293" exitCode=0
Jul 20 10:15:46.610692 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:46.610296    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" event={"ID":"60889480-945c-43a5-bf90-accfb84f6997","Type":"ContainerDied","Data":"c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293"}
Jul 20 10:15:46.612875 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:46.612852    2555 generic.go:356] "Generic (PLEG): container finished" podID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerID="c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99" exitCode=0
Jul 20 10:15:46.612998 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:46.612977    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" event={"ID":"aef7e726-3136-4c35-9eed-567c438d3ec8","Type":"ContainerDied","Data":"c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99"}
Jul 20 10:15:47.619573 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:47.619537    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" event={"ID":"60889480-945c-43a5-bf90-accfb84f6997","Type":"ContainerStarted","Data":"9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba"}
Jul 20 10:15:47.620081 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:47.619609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" event={"ID":"60889480-945c-43a5-bf90-accfb84f6997","Type":"ContainerStarted","Data":"ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e"}
Jul 20 10:15:47.620081 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:47.619830    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:47.647225 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:47.647155    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podStartSLOduration=2.89267463 podStartE2EDuration="6.647136459s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.754461829s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:15:47.645995821 +0000 UTC m=+1120.696925176" watchObservedRunningTime="2026-07-20 10:15:47.647136459 +0000 UTC m=+1120.698065814"
Jul 20 10:15:48.624182 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:48.624143    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:48.625701 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:48.625675    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:15:49.172597 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:49.172540    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 20 10:15:49.177984 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:49.177940    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:15:49.178305 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:49.178278    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:15:49.628903 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:49.628861    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:15:51.937785 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:51.937753    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:52.038474 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:52.038440    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:54.172682 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:54.172636    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 20 10:15:54.173079 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:54.172788    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:15:54.633816 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:54.633788    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:15:54.634320 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:54.634297    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:15:59.173071 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:59.173028    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 20 10:15:59.178520 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:59.178487    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:15:59.178842 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:15:59.178817    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:16:01.918418 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:01.918383    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:16:02.082165 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:02.082132    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:16:04.173481 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:04.173432    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 20 10:16:04.634667 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:04.634619    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:16:09.173532 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:09.173477    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 20 10:16:09.177907 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:09.177864    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:16:09.178074 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:09.178055    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:16:09.178275 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:09.178230    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:16:09.178377 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:09.178363    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:16:11.511975 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.511947    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:16:11.539346 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.539311    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aef7e726-3136-4c35-9eed-567c438d3ec8-proxy-tls\") pod \"aef7e726-3136-4c35-9eed-567c438d3ec8\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") "
Jul 20 10:16:11.539546 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.539366    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aef7e726-3136-4c35-9eed-567c438d3ec8-kserve-provision-location\") pod \"aef7e726-3136-4c35-9eed-567c438d3ec8\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") "
Jul 20 10:16:11.539546 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.539405    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-rbac-proxy-sar-config\") pod \"aef7e726-3136-4c35-9eed-567c438d3ec8\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") "
Jul 20 10:16:11.539546 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.539424    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zhjs\" (UniqueName: \"kubernetes.io/projected/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-api-access-5zhjs\") pod \"aef7e726-3136-4c35-9eed-567c438d3ec8\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") "
Jul 20 10:16:11.539546 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.539453    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-cabundle-cert\") pod \"aef7e726-3136-4c35-9eed-567c438d3ec8\" (UID: \"aef7e726-3136-4c35-9eed-567c438d3ec8\") "
Jul 20 10:16:11.539786 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.539755    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef7e726-3136-4c35-9eed-567c438d3ec8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aef7e726-3136-4c35-9eed-567c438d3ec8" (UID: "aef7e726-3136-4c35-9eed-567c438d3ec8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:16:11.540036 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.540012    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aef7e726-3136-4c35-9eed-567c438d3ec8" (UID: "aef7e726-3136-4c35-9eed-567c438d3ec8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:16:11.540165 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.540041    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aef7e726-3136-4c35-9eed-567c438d3ec8" (UID: "aef7e726-3136-4c35-9eed-567c438d3ec8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:16:11.541852 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.541821    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-api-access-5zhjs" (OuterVolumeSpecName: "kube-api-access-5zhjs") pod "aef7e726-3136-4c35-9eed-567c438d3ec8" (UID: "aef7e726-3136-4c35-9eed-567c438d3ec8"). InnerVolumeSpecName "kube-api-access-5zhjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:16:11.542022 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.542000    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef7e726-3136-4c35-9eed-567c438d3ec8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aef7e726-3136-4c35-9eed-567c438d3ec8" (UID: "aef7e726-3136-4c35-9eed-567c438d3ec8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:16:11.640670 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.640628    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aef7e726-3136-4c35-9eed-567c438d3ec8-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:16:11.640670 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.640662    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aef7e726-3136-4c35-9eed-567c438d3ec8-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:16:11.640670 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.640676    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:16:11.640928 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.640688    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhjs\" (UniqueName: \"kubernetes.io/projected/aef7e726-3136-4c35-9eed-567c438d3ec8-kube-api-access-5zhjs\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:16:11.640928 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.640701    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aef7e726-3136-4c35-9eed-567c438d3ec8-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:16:11.737660 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.737625    2555 generic.go:356] "Generic (PLEG): container finished" podID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerID="b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378" exitCode=0
Jul 20 10:16:11.737824 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.737689    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"
Jul 20 10:16:11.737824 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.737688    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" event={"ID":"aef7e726-3136-4c35-9eed-567c438d3ec8","Type":"ContainerDied","Data":"b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378"}
Jul 20 10:16:11.737904 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.737827    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg" event={"ID":"aef7e726-3136-4c35-9eed-567c438d3ec8","Type":"ContainerDied","Data":"16f5b46c2318334ac2867feb90053b1d21752f52800349270eb5df3f79453916"}
Jul 20 10:16:11.737904 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.737861    2555 scope.go:117] "RemoveContainer" containerID="b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378"
Jul 20 10:16:11.748150 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.748122    2555 scope.go:117] "RemoveContainer" containerID="9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f"
Jul 20 10:16:11.759381 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.759358    2555 scope.go:117] "RemoveContainer" containerID="c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99"
Jul 20 10:16:11.763579 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.763548    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"]
Jul 20 10:16:11.768359 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.768335    2555 scope.go:117] "RemoveContainer" containerID="6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3"
Jul 20 10:16:11.768707 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.768680    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22c6a-predictor-7c89f597df-4f4mg"]
Jul 20 10:16:11.776678 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.776658    2555 scope.go:117] "RemoveContainer" containerID="b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378"
Jul 20 10:16:11.777003 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:16:11.776981    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378\": container with ID starting with b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378 not found: ID does not exist" containerID="b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378"
Jul 20 10:16:11.777048 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.777015    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378"} err="failed to get container status \"b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378\": rpc error: code = NotFound desc = could not find container \"b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378\": container with ID starting with b815dfe2d237708472ae37921dfd37962fab81a4c1722337603e5442a7c4a378 not found: ID does not exist"
Jul 20 10:16:11.777048 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.777037    2555 scope.go:117] "RemoveContainer" containerID="9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f"
Jul 20 10:16:11.777318 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:16:11.777299    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f\": container with ID starting with 9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f not found: ID does not exist" containerID="9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f"
Jul 20 10:16:11.777391 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.777324    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f"} err="failed to get container status \"9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f\": rpc error: code = NotFound desc = could not find container \"9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f\": container with ID starting with 9b648456357497d985a59c33ece41c8ae20c7c8421a0c8c6e7f81ec0e998de0f not found: ID does not exist"
Jul 20 10:16:11.777391 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.777341    2555 scope.go:117] "RemoveContainer" containerID="c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99"
Jul 20 10:16:11.777579 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:16:11.777564    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99\": container with ID starting with c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99 not found: ID does not exist" containerID="c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99"
Jul 20 10:16:11.777619 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.777582    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99"} err="failed to get container status \"c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99\": rpc error: code = NotFound desc = could not find container \"c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99\": container with ID starting with c36cfe568e7d4ea5376f0d3f6cc98112427eb701a5b0c32276fc5f3bcdb58b99 not found: ID does not exist"
Jul 20 10:16:11.777619 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.777595    2555 scope.go:117] "RemoveContainer" containerID="6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3"
Jul 20 10:16:11.777828 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:16:11.777813    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3\": container with ID starting with 6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3 not found: ID does not exist" containerID="6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3"
Jul 20 10:16:11.777864 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:11.777832    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3"} err="failed to get container status \"6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3\": rpc error: code = NotFound desc = could not find container \"6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3\": container with ID starting with 6f7114aba4b3aaf5611de116580462e6d37df5f480e8b5752d36ad1df1b1e6e3 not found: ID does not exist"
Jul 20 10:16:13.496319 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:13.496288    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" path="/var/lib/kubelet/pods/aef7e726-3136-4c35-9eed-567c438d3ec8/volumes"
Jul 20 10:16:14.634925 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:14.634887    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:16:24.634233 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:24.634189    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:16:34.635073 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:34.635031    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:16:44.634844 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:44.634799    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:16:54.634776 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:16:54.634731    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:17:02.486750 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:17:02.486703    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:17:07.492485 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:17:07.492456    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:17:07.494408 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:17:07.494383    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:17:12.486791 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:17:12.486743    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:17:22.487197 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:17:22.487154    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:17:32.487659 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:17:32.487612    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:17:42.487460 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:17:42.487418    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:17:52.487489 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:17:52.487443    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:18:02.486988 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:02.486896    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:18:11.486727 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:11.486682    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:18:21.486626 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:21.486582    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:18:31.491590 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:31.491562    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:18:41.437992 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:41.437957    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"]
Jul 20 10:18:41.438455 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:41.438241    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" containerID="cri-o://ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e" gracePeriod=30
Jul 20 10:18:41.438455 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:41.438269    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kube-rbac-proxy" containerID="cri-o://9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba" gracePeriod=30
Jul 20 10:18:41.485219 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:41.485186    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:41.486573 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:41.486542    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:18:42.426136 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:42.426102    2555 generic.go:356] "Generic (PLEG): container finished" podID="60889480-945c-43a5-bf90-accfb84f6997" containerID="9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba" exitCode=2
Jul 20 10:18:42.426339 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:42.426160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" event={"ID":"60889480-945c-43a5-bf90-accfb84f6997","Type":"ContainerDied","Data":"9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba"}
Jul 20 10:18:44.630031 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:44.629988    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 20 10:18:46.550582 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.550544    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:46.607443 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607404    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"]
Jul 20 10:18:46.607805 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607792    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51afa2e-5ce2-42d9-99bc-30bda0468137" containerName="kserve-container"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607806    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51afa2e-5ce2-42d9-99bc-30bda0468137" containerName="kserve-container"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607813    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51afa2e-5ce2-42d9-99bc-30bda0468137" containerName="kube-rbac-proxy"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607819    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51afa2e-5ce2-42d9-99bc-30bda0468137" containerName="kube-rbac-proxy"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607827    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="storage-initializer"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607833    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="storage-initializer"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607840    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607845    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607855    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607860    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607868    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kube-rbac-proxy"
Jul 20 10:18:46.607901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607873    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kube-rbac-proxy"
Jul 20 10:18:46.608304 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607922    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kube-rbac-proxy"
Jul 20 10:18:46.608304 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607931    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51afa2e-5ce2-42d9-99bc-30bda0468137" containerName="kube-rbac-proxy"
Jul 20 10:18:46.608304 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607939    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="agent"
Jul 20 10:18:46.608304 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607944    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51afa2e-5ce2-42d9-99bc-30bda0468137" containerName="kserve-container"
Jul 20 10:18:46.608304 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.607952    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef7e726-3136-4c35-9eed-567c438d3ec8" containerName="kserve-container"
Jul 20 10:18:46.611308 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.611293    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.614090 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.614070    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cefa95-predictor-serving-cert\""
Jul 20 10:18:46.614335 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.614319    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cefa95-kube-rbac-proxy-sar-config\""
Jul 20 10:18:46.629775 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.629736    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"]
Jul 20 10:18:46.773443 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.773403    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kserve-provision-location\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.773634 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.773471    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-cabundle-cert\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.773634 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.773494    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r62m\" (UniqueName: \"kubernetes.io/projected/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-api-access-8r62m\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.773634 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.773515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.773634 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.773544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-proxy-tls\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.790154 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.790122    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:46.874350 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.874313    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-cabundle-cert\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.874503 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.874361    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r62m\" (UniqueName: \"kubernetes.io/projected/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-api-access-8r62m\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.874548 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.874496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.874590 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.874551    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-proxy-tls\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.874633 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.874598    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kserve-provision-location\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.874948 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.874926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kserve-provision-location\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.875049 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.875033    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-cabundle-cert\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.875116 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.875100    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.876996 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.876975    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-proxy-tls\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.884223 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.884201    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r62m\" (UniqueName: \"kubernetes.io/projected/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-api-access-8r62m\") pod \"isvc-primary-cefa95-predictor-5866df9cfc-2vppr\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") " pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:46.921763 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:46.921729    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:47.073064 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:47.073040    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"]
Jul 20 10:18:47.075240 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:18:47.075212    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966b0087_3b89_4a89_9a1d_11b3c7a28d0d.slice/crio-5f870cecb7e9202842089daf9d15be9b51d2f1cf66c3f7bacee9c5a85a15b660 WatchSource:0}: Error finding container 5f870cecb7e9202842089daf9d15be9b51d2f1cf66c3f7bacee9c5a85a15b660: Status 404 returned error can't find the container with id 5f870cecb7e9202842089daf9d15be9b51d2f1cf66c3f7bacee9c5a85a15b660
Jul 20 10:18:47.448599 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:47.448512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" event={"ID":"966b0087-3b89-4a89-9a1d-11b3c7a28d0d","Type":"ContainerStarted","Data":"7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2"}
Jul 20 10:18:47.448599 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:47.448578    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" event={"ID":"966b0087-3b89-4a89-9a1d-11b3c7a28d0d","Type":"ContainerStarted","Data":"5f870cecb7e9202842089daf9d15be9b51d2f1cf66c3f7bacee9c5a85a15b660"}
Jul 20 10:18:49.629143 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:49.629104    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 20 10:18:51.467961 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.467927    2555 generic.go:356] "Generic (PLEG): container finished" podID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerID="7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2" exitCode=0
Jul 20 10:18:51.468351 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.467995    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" event={"ID":"966b0087-3b89-4a89-9a1d-11b3c7a28d0d","Type":"ContainerDied","Data":"7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2"}
Jul 20 10:18:51.487117 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.487078    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:18:51.801461 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.801436    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:18:51.920059 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.920018    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfjw7\" (UniqueName: \"kubernetes.io/projected/60889480-945c-43a5-bf90-accfb84f6997-kube-api-access-mfjw7\") pod \"60889480-945c-43a5-bf90-accfb84f6997\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") "
Jul 20 10:18:51.920059 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.920065    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60889480-945c-43a5-bf90-accfb84f6997-kserve-provision-location\") pod \"60889480-945c-43a5-bf90-accfb84f6997\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") "
Jul 20 10:18:51.920330 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.920097    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60889480-945c-43a5-bf90-accfb84f6997-proxy-tls\") pod \"60889480-945c-43a5-bf90-accfb84f6997\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") "
Jul 20 10:18:51.920330 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.920165    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-kube-rbac-proxy-sar-config\") pod \"60889480-945c-43a5-bf90-accfb84f6997\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") "
Jul 20 10:18:51.920330 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.920189    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-cabundle-cert\") pod \"60889480-945c-43a5-bf90-accfb84f6997\" (UID: \"60889480-945c-43a5-bf90-accfb84f6997\") "
Jul 20 10:18:51.920493 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.920462    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60889480-945c-43a5-bf90-accfb84f6997-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60889480-945c-43a5-bf90-accfb84f6997" (UID: "60889480-945c-43a5-bf90-accfb84f6997"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:51.920618 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.920590    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "60889480-945c-43a5-bf90-accfb84f6997" (UID: "60889480-945c-43a5-bf90-accfb84f6997"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:18:51.920673 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.920612    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "60889480-945c-43a5-bf90-accfb84f6997" (UID: "60889480-945c-43a5-bf90-accfb84f6997"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:18:51.922345 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.922320    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60889480-945c-43a5-bf90-accfb84f6997-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "60889480-945c-43a5-bf90-accfb84f6997" (UID: "60889480-945c-43a5-bf90-accfb84f6997"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:18:51.922468 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:51.922451    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60889480-945c-43a5-bf90-accfb84f6997-kube-api-access-mfjw7" (OuterVolumeSpecName: "kube-api-access-mfjw7") pod "60889480-945c-43a5-bf90-accfb84f6997" (UID: "60889480-945c-43a5-bf90-accfb84f6997"). InnerVolumeSpecName "kube-api-access-mfjw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:18:52.021091 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.021049    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:18:52.021091 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.021082    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60889480-945c-43a5-bf90-accfb84f6997-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:18:52.021091 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.021091    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfjw7\" (UniqueName: \"kubernetes.io/projected/60889480-945c-43a5-bf90-accfb84f6997-kube-api-access-mfjw7\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:18:52.021091 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.021100    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60889480-945c-43a5-bf90-accfb84f6997-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:18:52.021383 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.021110    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60889480-945c-43a5-bf90-accfb84f6997-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:18:52.474428 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.474388    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" event={"ID":"966b0087-3b89-4a89-9a1d-11b3c7a28d0d","Type":"ContainerStarted","Data":"eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483"}
Jul 20 10:18:52.474864 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.474459    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" event={"ID":"966b0087-3b89-4a89-9a1d-11b3c7a28d0d","Type":"ContainerStarted","Data":"bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13"}
Jul 20 10:18:52.474864 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.474672    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:52.476169 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.476144    2555 generic.go:356] "Generic (PLEG): container finished" podID="60889480-945c-43a5-bf90-accfb84f6997" containerID="ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e" exitCode=0
Jul 20 10:18:52.476290 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.476203    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"
Jul 20 10:18:52.476290 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.476222    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" event={"ID":"60889480-945c-43a5-bf90-accfb84f6997","Type":"ContainerDied","Data":"ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e"}
Jul 20 10:18:52.476290 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.476284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n" event={"ID":"60889480-945c-43a5-bf90-accfb84f6997","Type":"ContainerDied","Data":"2cbbf8862fcca6f66cff85bfc8044fea6bd664c15f4a827a2281c517270c57ec"}
Jul 20 10:18:52.476396 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.476311    2555 scope.go:117] "RemoveContainer" containerID="9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba"
Jul 20 10:18:52.484773 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.484757    2555 scope.go:117] "RemoveContainer" containerID="ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e"
Jul 20 10:18:52.492463 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.492441    2555 scope.go:117] "RemoveContainer" containerID="c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293"
Jul 20 10:18:52.500480 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.500460    2555 scope.go:117] "RemoveContainer" containerID="9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba"
Jul 20 10:18:52.500762 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:18:52.500738    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba\": container with ID starting with 9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba not found: ID does not exist" containerID="9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba"
Jul 20 10:18:52.500821 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.500776    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba"} err="failed to get container status \"9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba\": rpc error: code = NotFound desc = could not find container \"9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba\": container with ID starting with 9c68b59dc207945b494443dbc8b96ddade6b51575c64117d8bcac65f7423f0ba not found: ID does not exist"
Jul 20 10:18:52.500821 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.500804    2555 scope.go:117] "RemoveContainer" containerID="ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e"
Jul 20 10:18:52.501077 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:18:52.501054    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e\": container with ID starting with ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e not found: ID does not exist" containerID="ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e"
Jul 20 10:18:52.501151 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.501090    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e"} err="failed to get container status \"ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e\": rpc error: code = NotFound desc = could not find container \"ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e\": container with ID starting with ec27e10d67077908d5601328f76ac478e86dd1c3d7b84585f33f1c221d0cd43e not found: ID does not exist"
Jul 20 10:18:52.501151 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.501108    2555 scope.go:117] "RemoveContainer" containerID="c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293"
Jul 20 10:18:52.501402 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:18:52.501380    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293\": container with ID starting with c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293 not found: ID does not exist" containerID="c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293"
Jul 20 10:18:52.501503 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.501405    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293"} err="failed to get container status \"c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293\": rpc error: code = NotFound desc = could not find container \"c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293\": container with ID starting with c5f445316bab1cea431066a11eb18d3abaec59d8c0dcabf28db9ffc7c57f3293 not found: ID does not exist"
Jul 20 10:18:52.514827 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.514762    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podStartSLOduration=2.613395206 podStartE2EDuration="6.514739752s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.901344546s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:18:52.501295548 +0000 UTC m=+1305.552224904" watchObservedRunningTime="2026-07-20 10:18:52.514739752 +0000 UTC m=+1305.565669108"
Jul 20 10:18:52.515941 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.515918    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"]
Jul 20 10:18:52.521376 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:52.521347    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-07030-predictor-888cc9fdb-pp67n"]
Jul 20 10:18:53.482918 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:53.482885    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:53.484495 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:53.484465    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:18:53.490487 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:53.490462    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60889480-945c-43a5-bf90-accfb84f6997" path="/var/lib/kubelet/pods/60889480-945c-43a5-bf90-accfb84f6997/volumes"
Jul 20 10:18:54.488141 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:54.488096    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:18:59.493220 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:59.493188    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:18:59.493700 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:18:59.493671    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:19:09.494315 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:19:09.494278    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:19:19.494580 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:19:19.494543    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:19:29.494315 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:19:29.494201    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:19:39.494483 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:19:39.494438    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:19:49.493894 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:19:49.493853    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:19:59.494435 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:19:59.494391    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:20:04.486351 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:04.486307    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:20:14.487470 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:14.487439    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:20:31.963347 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.963311    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"]
Jul 20 10:20:31.963765 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.963710    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container"
Jul 20 10:20:31.963765 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.963722    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container"
Jul 20 10:20:31.963765 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.963732    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="storage-initializer"
Jul 20 10:20:31.963765 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.963737    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="storage-initializer"
Jul 20 10:20:31.963765 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.963746    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kube-rbac-proxy"
Jul 20 10:20:31.963765 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.963753    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kube-rbac-proxy"
Jul 20 10:20:31.963954 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.963810    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kserve-container"
Jul 20 10:20:31.963954 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.963820    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="60889480-945c-43a5-bf90-accfb84f6997" containerName="kube-rbac-proxy"
Jul 20 10:20:31.967312 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.967282    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:31.967792 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.967764    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:31.970034 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.970010    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-cefa95\""
Jul 20 10:20:31.970150 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.970032    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cefa95-kube-rbac-proxy-sar-config\""
Jul 20 10:20:31.970150 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.970076    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cefa95-predictor-serving-cert\""
Jul 20 10:20:31.970150 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.970010    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-cefa95-dockercfg-kcg4h\""
Jul 20 10:20:31.978705 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:31.978680    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"]
Jul 20 10:20:32.048843 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.048801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da29b45a-e485-4fa7-b8cc-d4c004062292-proxy-tls\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.048843 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.048841    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqz8r\" (UniqueName: \"kubernetes.io/projected/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-api-access-rqz8r\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.049068 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.048912    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da29b45a-e485-4fa7-b8cc-d4c004062292-kserve-provision-location\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.049068 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.048955    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-cabundle-cert\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.049068 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.049015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.051218 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.051196    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:32.150312 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.150236    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da29b45a-e485-4fa7-b8cc-d4c004062292-proxy-tls\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.150312 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.150309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqz8r\" (UniqueName: \"kubernetes.io/projected/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-api-access-rqz8r\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.150584 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.150353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da29b45a-e485-4fa7-b8cc-d4c004062292-kserve-provision-location\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.150584 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.150376    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-cabundle-cert\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.150584 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.150434    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.150833 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.150807    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da29b45a-e485-4fa7-b8cc-d4c004062292-kserve-provision-location\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.151260 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.151225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.151318 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.151225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-cabundle-cert\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.152836 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.152811    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da29b45a-e485-4fa7-b8cc-d4c004062292-proxy-tls\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.159875 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.159844    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqz8r\" (UniqueName: \"kubernetes.io/projected/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-api-access-rqz8r\") pod \"isvc-secondary-cefa95-predictor-664c5956d6-h5cvl\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") " pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.278602 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.278504    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:32.420426 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.420391    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"]
Jul 20 10:20:32.421420 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:20:32.421391    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda29b45a_e485_4fa7_b8cc_d4c004062292.slice/crio-d66b1df458f9274ab5039c5f62314ecd25bc2b70bd56507b1d58e008cec3301d WatchSource:0}: Error finding container d66b1df458f9274ab5039c5f62314ecd25bc2b70bd56507b1d58e008cec3301d: Status 404 returned error can't find the container with id d66b1df458f9274ab5039c5f62314ecd25bc2b70bd56507b1d58e008cec3301d
Jul 20 10:20:32.930382 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.930344    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl" event={"ID":"da29b45a-e485-4fa7-b8cc-d4c004062292","Type":"ContainerStarted","Data":"372ef59ef9e9dc65f394f98bbf1ac51d2b3ee52e085f87eea230f7aadfac2d32"}
Jul 20 10:20:32.930578 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:32.930410    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl" event={"ID":"da29b45a-e485-4fa7-b8cc-d4c004062292","Type":"ContainerStarted","Data":"d66b1df458f9274ab5039c5f62314ecd25bc2b70bd56507b1d58e008cec3301d"}
Jul 20 10:20:38.956991 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:38.956960    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cefa95-predictor-664c5956d6-h5cvl_da29b45a-e485-4fa7-b8cc-d4c004062292/storage-initializer/0.log"
Jul 20 10:20:38.957447 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:38.957011    2555 generic.go:356] "Generic (PLEG): container finished" podID="da29b45a-e485-4fa7-b8cc-d4c004062292" containerID="372ef59ef9e9dc65f394f98bbf1ac51d2b3ee52e085f87eea230f7aadfac2d32" exitCode=1
Jul 20 10:20:38.957447 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:38.957099    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl" event={"ID":"da29b45a-e485-4fa7-b8cc-d4c004062292","Type":"ContainerDied","Data":"372ef59ef9e9dc65f394f98bbf1ac51d2b3ee52e085f87eea230f7aadfac2d32"}
Jul 20 10:20:39.963750 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:39.963717    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cefa95-predictor-664c5956d6-h5cvl_da29b45a-e485-4fa7-b8cc-d4c004062292/storage-initializer/0.log"
Jul 20 10:20:39.964181 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:39.963838    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl" event={"ID":"da29b45a-e485-4fa7-b8cc-d4c004062292","Type":"ContainerStarted","Data":"df7af61a741de660bd8fb6284c8300b45156075a2f371ddb95f07ad6cffb9a84"}
Jul 20 10:20:44.987958 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:44.987929    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cefa95-predictor-664c5956d6-h5cvl_da29b45a-e485-4fa7-b8cc-d4c004062292/storage-initializer/1.log"
Jul 20 10:20:44.988372 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:44.988345    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cefa95-predictor-664c5956d6-h5cvl_da29b45a-e485-4fa7-b8cc-d4c004062292/storage-initializer/0.log"
Jul 20 10:20:44.988430 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:44.988376    2555 generic.go:356] "Generic (PLEG): container finished" podID="da29b45a-e485-4fa7-b8cc-d4c004062292" containerID="df7af61a741de660bd8fb6284c8300b45156075a2f371ddb95f07ad6cffb9a84" exitCode=1
Jul 20 10:20:44.988478 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:44.988456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl" event={"ID":"da29b45a-e485-4fa7-b8cc-d4c004062292","Type":"ContainerDied","Data":"df7af61a741de660bd8fb6284c8300b45156075a2f371ddb95f07ad6cffb9a84"}
Jul 20 10:20:44.988536 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:44.988525    2555 scope.go:117] "RemoveContainer" containerID="372ef59ef9e9dc65f394f98bbf1ac51d2b3ee52e085f87eea230f7aadfac2d32"
Jul 20 10:20:44.988880 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:44.988863    2555 scope.go:117] "RemoveContainer" containerID="372ef59ef9e9dc65f394f98bbf1ac51d2b3ee52e085f87eea230f7aadfac2d32"
Jul 20 10:20:44.999299 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:20:44.999269    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-cefa95-predictor-664c5956d6-h5cvl_kserve-ci-e2e-test(da29b45a-e485-4fa7-b8cc-d4c004062292)\"" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl" podUID="da29b45a-e485-4fa7-b8cc-d4c004062292"
Jul 20 10:20:44.999483 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:20:44.999450    2555 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cefa95-predictor-664c5956d6-h5cvl_kserve-ci-e2e-test_da29b45a-e485-4fa7-b8cc-d4c004062292_0 in pod sandbox d66b1df458f9274ab5039c5f62314ecd25bc2b70bd56507b1d58e008cec3301d from index: no such id: '372ef59ef9e9dc65f394f98bbf1ac51d2b3ee52e085f87eea230f7aadfac2d32'" containerID="372ef59ef9e9dc65f394f98bbf1ac51d2b3ee52e085f87eea230f7aadfac2d32"
Jul 20 10:20:44.999537 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:44.999496    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372ef59ef9e9dc65f394f98bbf1ac51d2b3ee52e085f87eea230f7aadfac2d32"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cefa95-predictor-664c5956d6-h5cvl_kserve-ci-e2e-test_da29b45a-e485-4fa7-b8cc-d4c004062292_0 in pod sandbox d66b1df458f9274ab5039c5f62314ecd25bc2b70bd56507b1d58e008cec3301d from index: no such id: '372ef59ef9e9dc65f394f98bbf1ac51d2b3ee52e085f87eea230f7aadfac2d32'"
Jul 20 10:20:45.994210 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:45.994183    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cefa95-predictor-664c5956d6-h5cvl_da29b45a-e485-4fa7-b8cc-d4c004062292/storage-initializer/1.log"
Jul 20 10:20:49.932634 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:49.932592    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"]
Jul 20 10:20:49.933039 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:49.932907    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" containerID="cri-o://bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13" gracePeriod=30
Jul 20 10:20:49.933111 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:49.933018    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kube-rbac-proxy" containerID="cri-o://eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483" gracePeriod=30
Jul 20 10:20:49.960318 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:49.960235    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:49.993684 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:49.993649    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"]
Jul 20 10:20:50.130804 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.130779    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cefa95-predictor-664c5956d6-h5cvl_da29b45a-e485-4fa7-b8cc-d4c004062292/storage-initializer/1.log"
Jul 20 10:20:50.133122 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.133102    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:50.215968 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.215876    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-cabundle-cert\") pod \"da29b45a-e485-4fa7-b8cc-d4c004062292\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") "
Jul 20 10:20:50.215968 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.215934    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da29b45a-e485-4fa7-b8cc-d4c004062292-proxy-tls\") pod \"da29b45a-e485-4fa7-b8cc-d4c004062292\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") "
Jul 20 10:20:50.215968 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.215970    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-rbac-proxy-sar-config\") pod \"da29b45a-e485-4fa7-b8cc-d4c004062292\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") "
Jul 20 10:20:50.216213 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.215993    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqz8r\" (UniqueName: \"kubernetes.io/projected/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-api-access-rqz8r\") pod \"da29b45a-e485-4fa7-b8cc-d4c004062292\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") "
Jul 20 10:20:50.216213 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.216065    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da29b45a-e485-4fa7-b8cc-d4c004062292-kserve-provision-location\") pod \"da29b45a-e485-4fa7-b8cc-d4c004062292\" (UID: \"da29b45a-e485-4fa7-b8cc-d4c004062292\") "
Jul 20 10:20:50.216380 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.216351    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da29b45a-e485-4fa7-b8cc-d4c004062292" (UID: "da29b45a-e485-4fa7-b8cc-d4c004062292"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:20:50.216453 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.216364    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da29b45a-e485-4fa7-b8cc-d4c004062292" (UID: "da29b45a-e485-4fa7-b8cc-d4c004062292"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:20:50.216453 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.216372    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da29b45a-e485-4fa7-b8cc-d4c004062292-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da29b45a-e485-4fa7-b8cc-d4c004062292" (UID: "da29b45a-e485-4fa7-b8cc-d4c004062292"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:50.218142 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.218111    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-api-access-rqz8r" (OuterVolumeSpecName: "kube-api-access-rqz8r") pod "da29b45a-e485-4fa7-b8cc-d4c004062292" (UID: "da29b45a-e485-4fa7-b8cc-d4c004062292"). InnerVolumeSpecName "kube-api-access-rqz8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:20:50.218293 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.218199    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da29b45a-e485-4fa7-b8cc-d4c004062292-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da29b45a-e485-4fa7-b8cc-d4c004062292" (UID: "da29b45a-e485-4fa7-b8cc-d4c004062292"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:20:50.317275 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.317214    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da29b45a-e485-4fa7-b8cc-d4c004062292-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:20:50.317275 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.317276    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:20:50.317275 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.317286    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqz8r\" (UniqueName: \"kubernetes.io/projected/da29b45a-e485-4fa7-b8cc-d4c004062292-kube-api-access-rqz8r\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:20:50.317581 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.317297    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da29b45a-e485-4fa7-b8cc-d4c004062292-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:20:50.317581 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:50.317306    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da29b45a-e485-4fa7-b8cc-d4c004062292-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:20:51.020363 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:51.020332    2555 generic.go:356] "Generic (PLEG): container finished" podID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerID="eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483" exitCode=2
Jul 20 10:20:51.020818 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:51.020407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" event={"ID":"966b0087-3b89-4a89-9a1d-11b3c7a28d0d","Type":"ContainerDied","Data":"eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483"}
Jul 20 10:20:51.021704 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:51.021685    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cefa95-predictor-664c5956d6-h5cvl_da29b45a-e485-4fa7-b8cc-d4c004062292/storage-initializer/1.log"
Jul 20 10:20:51.021802 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:51.021725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl" event={"ID":"da29b45a-e485-4fa7-b8cc-d4c004062292","Type":"ContainerDied","Data":"d66b1df458f9274ab5039c5f62314ecd25bc2b70bd56507b1d58e008cec3301d"}
Jul 20 10:20:51.021802 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:51.021771    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"
Jul 20 10:20:51.021871 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:51.021776    2555 scope.go:117] "RemoveContainer" containerID="df7af61a741de660bd8fb6284c8300b45156075a2f371ddb95f07ad6cffb9a84"
Jul 20 10:20:51.073707 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:51.073670    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"]
Jul 20 10:20:51.075613 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:51.075585    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cefa95-predictor-664c5956d6-h5cvl"]
Jul 20 10:20:51.429375 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:51.429329    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:51.491893 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:51.491858    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da29b45a-e485-4fa7-b8cc-d4c004062292" path="/var/lib/kubelet/pods/da29b45a-e485-4fa7-b8cc-d4c004062292/volumes"
Jul 20 10:20:54.486476 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:54.486429    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:20:54.488849 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:54.488822    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 20 10:20:55.176847 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.176814    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:55.194947 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.194919    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:20:55.213814 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.213721    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"]
Jul 20 10:20:55.214084 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214070    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container"
Jul 20 10:20:55.214131 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214086    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container"
Jul 20 10:20:55.214131 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214099    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kube-rbac-proxy"
Jul 20 10:20:55.214131 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214105    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kube-rbac-proxy"
Jul 20 10:20:55.214131 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214116    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="storage-initializer"
Jul 20 10:20:55.214131 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214122    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="storage-initializer"
Jul 20 10:20:55.214131 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214127    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da29b45a-e485-4fa7-b8cc-d4c004062292" containerName="storage-initializer"
Jul 20 10:20:55.214131 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214133    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="da29b45a-e485-4fa7-b8cc-d4c004062292" containerName="storage-initializer"
Jul 20 10:20:55.214401 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214203    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="da29b45a-e485-4fa7-b8cc-d4c004062292" containerName="storage-initializer"
Jul 20 10:20:55.214401 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214214    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="da29b45a-e485-4fa7-b8cc-d4c004062292" containerName="storage-initializer"
Jul 20 10:20:55.214401 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214222    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kube-rbac-proxy"
Jul 20 10:20:55.214401 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214233    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerName="kserve-container"
Jul 20 10:20:55.214401 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214334    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da29b45a-e485-4fa7-b8cc-d4c004062292" containerName="storage-initializer"
Jul 20 10:20:55.214401 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.214342    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="da29b45a-e485-4fa7-b8cc-d4c004062292" containerName="storage-initializer"
Jul 20 10:20:55.217679 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.217647    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.220850 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.220823    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-531344-predictor-serving-cert\""
Jul 20 10:20:55.221022 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.220877    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-531344-dockercfg-5x9h6\""
Jul 20 10:20:55.221022 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.220898    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-531344\""
Jul 20 10:20:55.221129 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.221024    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-531344-kube-rbac-proxy-sar-config\""
Jul 20 10:20:55.230565 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.230527    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"]
Jul 20 10:20:55.363655 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.363611    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-rbac-proxy-sar-config\") pod \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") "
Jul 20 10:20:55.363843 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.363677    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-proxy-tls\") pod \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") "
Jul 20 10:20:55.363843 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.363720    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-cabundle-cert\") pod \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") "
Jul 20 10:20:55.363843 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.363792    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kserve-provision-location\") pod \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") "
Jul 20 10:20:55.364024 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.363856    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r62m\" (UniqueName: \"kubernetes.io/projected/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-api-access-8r62m\") pod \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\" (UID: \"966b0087-3b89-4a89-9a1d-11b3c7a28d0d\") "
Jul 20 10:20:55.364024 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364013    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d473c2-c721-4e08-aed3-c70db8a7b969-kserve-provision-location\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.364130 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364043    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "966b0087-3b89-4a89-9a1d-11b3c7a28d0d" (UID: "966b0087-3b89-4a89-9a1d-11b3c7a28d0d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:20:55.364130 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364107    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "966b0087-3b89-4a89-9a1d-11b3c7a28d0d" (UID: "966b0087-3b89-4a89-9a1d-11b3c7a28d0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:55.364130 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364104    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "966b0087-3b89-4a89-9a1d-11b3c7a28d0d" (UID: "966b0087-3b89-4a89-9a1d-11b3c7a28d0d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:20:55.364292 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364134    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d473c2-c721-4e08-aed3-c70db8a7b969-proxy-tls\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.364292 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364194    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.364292 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364262    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvqf\" (UniqueName: \"kubernetes.io/projected/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-api-access-7vvqf\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.364292 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364287    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-cabundle-cert\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.364446 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364370    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:20:55.364446 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364388    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:20:55.364446 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.364403    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:20:55.365877 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.365850    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "966b0087-3b89-4a89-9a1d-11b3c7a28d0d" (UID: "966b0087-3b89-4a89-9a1d-11b3c7a28d0d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:20:55.365877 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.365853    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-api-access-8r62m" (OuterVolumeSpecName: "kube-api-access-8r62m") pod "966b0087-3b89-4a89-9a1d-11b3c7a28d0d" (UID: "966b0087-3b89-4a89-9a1d-11b3c7a28d0d"). InnerVolumeSpecName "kube-api-access-8r62m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:20:55.465128 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.465036    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d473c2-c721-4e08-aed3-c70db8a7b969-kserve-provision-location\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.465128 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.465100    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d473c2-c721-4e08-aed3-c70db8a7b969-proxy-tls\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.465380 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:20:55.465194    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-531344-predictor-serving-cert: secret "isvc-init-fail-531344-predictor-serving-cert" not found
Jul 20 10:20:55.465380 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.465211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.465380 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:20:55.465270    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75d473c2-c721-4e08-aed3-c70db8a7b969-proxy-tls podName:75d473c2-c721-4e08-aed3-c70db8a7b969 nodeName:}" failed. No retries permitted until 2026-07-20 10:20:55.965227669 +0000 UTC m=+1429.016157001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/75d473c2-c721-4e08-aed3-c70db8a7b969-proxy-tls") pod "isvc-init-fail-531344-predictor-684755d79b-ghbr5" (UID: "75d473c2-c721-4e08-aed3-c70db8a7b969") : secret "isvc-init-fail-531344-predictor-serving-cert" not found
Jul 20 10:20:55.465380 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.465300    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvqf\" (UniqueName: \"kubernetes.io/projected/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-api-access-7vvqf\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.465380 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.465319    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-cabundle-cert\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.465627 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.465405    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r62m\" (UniqueName: \"kubernetes.io/projected/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-kube-api-access-8r62m\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:20:55.465627 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.465421    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/966b0087-3b89-4a89-9a1d-11b3c7a28d0d-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:20:55.465627 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.465493    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d473c2-c721-4e08-aed3-c70db8a7b969-kserve-provision-location\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.465878 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.465860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.465878 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.465873    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-cabundle-cert\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.475530 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.475497    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvqf\" (UniqueName: \"kubernetes.io/projected/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-api-access-7vvqf\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.946600 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.946560    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:55.970547 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.970503    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d473c2-c721-4e08-aed3-c70db8a7b969-proxy-tls\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:55.973007 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:55.972973    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d473c2-c721-4e08-aed3-c70db8a7b969-proxy-tls\") pod \"isvc-init-fail-531344-predictor-684755d79b-ghbr5\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") " pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:56.047683 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.047646    2555 generic.go:356] "Generic (PLEG): container finished" podID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" containerID="bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13" exitCode=0
Jul 20 10:20:56.047822 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.047712    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"
Jul 20 10:20:56.047822 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.047726    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" event={"ID":"966b0087-3b89-4a89-9a1d-11b3c7a28d0d","Type":"ContainerDied","Data":"bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13"}
Jul 20 10:20:56.047822 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.047794    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr" event={"ID":"966b0087-3b89-4a89-9a1d-11b3c7a28d0d","Type":"ContainerDied","Data":"5f870cecb7e9202842089daf9d15be9b51d2f1cf66c3f7bacee9c5a85a15b660"}
Jul 20 10:20:56.047947 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.047823    2555 scope.go:117] "RemoveContainer" containerID="eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483"
Jul 20 10:20:56.056797 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.056597    2555 scope.go:117] "RemoveContainer" containerID="bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13"
Jul 20 10:20:56.064338 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.064309    2555 scope.go:117] "RemoveContainer" containerID="7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2"
Jul 20 10:20:56.070978 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.070949    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"]
Jul 20 10:20:56.073572 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.073551    2555 scope.go:117] "RemoveContainer" containerID="eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483"
Jul 20 10:20:56.073925 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:20:56.073902    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483\": container with ID starting with eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483 not found: ID does not exist" containerID="eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483"
Jul 20 10:20:56.074017 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.073936    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483"} err="failed to get container status \"eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483\": rpc error: code = NotFound desc = could not find container \"eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483\": container with ID starting with eb4216dfb60f6f17e2d282a463ec04e878950152a8f2329457056acb61015483 not found: ID does not exist"
Jul 20 10:20:56.074017 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.073957    2555 scope.go:117] "RemoveContainer" containerID="bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13"
Jul 20 10:20:56.074320 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:20:56.074291    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13\": container with ID starting with bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13 not found: ID does not exist" containerID="bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13"
Jul 20 10:20:56.074411 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.074330    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13"} err="failed to get container status \"bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13\": rpc error: code = NotFound desc = could not find container \"bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13\": container with ID starting with bd15f163da3aaccb96c7b7d6b7ef9919c934ab89e9170c1667d2b335b3b09b13 not found: ID does not exist"
Jul 20 10:20:56.074411 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.074352    2555 scope.go:117] "RemoveContainer" containerID="7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2"
Jul 20 10:20:56.074629 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:20:56.074610    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2\": container with ID starting with 7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2 not found: ID does not exist" containerID="7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2"
Jul 20 10:20:56.074681 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.074635    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2"} err="failed to get container status \"7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2\": rpc error: code = NotFound desc = could not find container \"7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2\": container with ID starting with 7aeb35016a57da63c492ea563e3e7be440df57f6656c219ba71fe60e9d40cef2 not found: ID does not exist"
Jul 20 10:20:56.075358 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.075339    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cefa95-predictor-5866df9cfc-2vppr"]
Jul 20 10:20:56.129554 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.129516    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:20:56.271126 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:56.271070    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"]
Jul 20 10:20:56.273866 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:20:56.273837    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d473c2_c721_4e08_aed3_c70db8a7b969.slice/crio-c42884e3354ddae2df3fd810932f22422b2dcc854d12c7965fb3a4b8d1dd8b79 WatchSource:0}: Error finding container c42884e3354ddae2df3fd810932f22422b2dcc854d12c7965fb3a4b8d1dd8b79: Status 404 returned error can't find the container with id c42884e3354ddae2df3fd810932f22422b2dcc854d12c7965fb3a4b8d1dd8b79
Jul 20 10:20:57.058972 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:57.058928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5" event={"ID":"75d473c2-c721-4e08-aed3-c70db8a7b969","Type":"ContainerStarted","Data":"ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1"}
Jul 20 10:20:57.059481 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:57.059041    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5" event={"ID":"75d473c2-c721-4e08-aed3-c70db8a7b969","Type":"ContainerStarted","Data":"c42884e3354ddae2df3fd810932f22422b2dcc854d12c7965fb3a4b8d1dd8b79"}
Jul 20 10:20:57.491478 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:20:57.491442    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966b0087-3b89-4a89-9a1d-11b3c7a28d0d" path="/var/lib/kubelet/pods/966b0087-3b89-4a89-9a1d-11b3c7a28d0d/volumes"
Jul 20 10:21:02.083571 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:02.083489    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-531344-predictor-684755d79b-ghbr5_75d473c2-c721-4e08-aed3-c70db8a7b969/storage-initializer/0.log"
Jul 20 10:21:02.083571 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:02.083528    2555 generic.go:356] "Generic (PLEG): container finished" podID="75d473c2-c721-4e08-aed3-c70db8a7b969" containerID="ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1" exitCode=1
Jul 20 10:21:02.084062 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:02.083604    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5" event={"ID":"75d473c2-c721-4e08-aed3-c70db8a7b969","Type":"ContainerDied","Data":"ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1"}
Jul 20 10:21:03.090417 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:03.090388    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-531344-predictor-684755d79b-ghbr5_75d473c2-c721-4e08-aed3-c70db8a7b969/storage-initializer/0.log"
Jul 20 10:21:03.090817 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:03.090472    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5" event={"ID":"75d473c2-c721-4e08-aed3-c70db8a7b969","Type":"ContainerStarted","Data":"3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb"}
Jul 20 10:21:05.213155 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:05.213112    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"]
Jul 20 10:21:05.213652 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:05.213396    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5" podUID="75d473c2-c721-4e08-aed3-c70db8a7b969" containerName="storage-initializer" containerID="cri-o://3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb" gracePeriod=30
Jul 20 10:21:05.241041 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:05.241000    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:06.863507 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.863482    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-531344-predictor-684755d79b-ghbr5_75d473c2-c721-4e08-aed3-c70db8a7b969/storage-initializer/1.log"
Jul 20 10:21:06.863929 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.863913    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-531344-predictor-684755d79b-ghbr5_75d473c2-c721-4e08-aed3-c70db8a7b969/storage-initializer/0.log"
Jul 20 10:21:06.866191 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.866174    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:21:06.962968 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.962873    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vvqf\" (UniqueName: \"kubernetes.io/projected/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-api-access-7vvqf\") pod \"75d473c2-c721-4e08-aed3-c70db8a7b969\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") "
Jul 20 10:21:06.962968 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.962914    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-rbac-proxy-sar-config\") pod \"75d473c2-c721-4e08-aed3-c70db8a7b969\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") "
Jul 20 10:21:06.962968 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.962959    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-cabundle-cert\") pod \"75d473c2-c721-4e08-aed3-c70db8a7b969\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") "
Jul 20 10:21:06.963302 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.963071    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d473c2-c721-4e08-aed3-c70db8a7b969-kserve-provision-location\") pod \"75d473c2-c721-4e08-aed3-c70db8a7b969\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") "
Jul 20 10:21:06.963302 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.963140    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d473c2-c721-4e08-aed3-c70db8a7b969-proxy-tls\") pod \"75d473c2-c721-4e08-aed3-c70db8a7b969\" (UID: \"75d473c2-c721-4e08-aed3-c70db8a7b969\") "
Jul 20 10:21:06.963414 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.963294    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75d473c2-c721-4e08-aed3-c70db8a7b969" (UID: "75d473c2-c721-4e08-aed3-c70db8a7b969"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:21:06.963414 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.963369    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d473c2-c721-4e08-aed3-c70db8a7b969-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75d473c2-c721-4e08-aed3-c70db8a7b969" (UID: "75d473c2-c721-4e08-aed3-c70db8a7b969"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:21:06.963414 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.963387    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75d473c2-c721-4e08-aed3-c70db8a7b969" (UID: "75d473c2-c721-4e08-aed3-c70db8a7b969"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:21:06.963560 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.963485    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:21:06.963560 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.963505    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d473c2-c721-4e08-aed3-c70db8a7b969-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:21:06.963560 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.963522    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:21:06.965229 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.965201    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d473c2-c721-4e08-aed3-c70db8a7b969-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75d473c2-c721-4e08-aed3-c70db8a7b969" (UID: "75d473c2-c721-4e08-aed3-c70db8a7b969"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:21:06.965386 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:06.965291    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-api-access-7vvqf" (OuterVolumeSpecName: "kube-api-access-7vvqf") pod "75d473c2-c721-4e08-aed3-c70db8a7b969" (UID: "75d473c2-c721-4e08-aed3-c70db8a7b969"). InnerVolumeSpecName "kube-api-access-7vvqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:21:07.064867 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.064820    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d473c2-c721-4e08-aed3-c70db8a7b969-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:21:07.064867 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.064860    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvqf\" (UniqueName: \"kubernetes.io/projected/75d473c2-c721-4e08-aed3-c70db8a7b969-kube-api-access-7vvqf\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:21:07.110570 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.110544    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-531344-predictor-684755d79b-ghbr5_75d473c2-c721-4e08-aed3-c70db8a7b969/storage-initializer/1.log"
Jul 20 10:21:07.110966 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.110947    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-531344-predictor-684755d79b-ghbr5_75d473c2-c721-4e08-aed3-c70db8a7b969/storage-initializer/0.log"
Jul 20 10:21:07.111014 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.110985    2555 generic.go:356] "Generic (PLEG): container finished" podID="75d473c2-c721-4e08-aed3-c70db8a7b969" containerID="3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb" exitCode=1
Jul 20 10:21:07.111049 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.111013    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5" event={"ID":"75d473c2-c721-4e08-aed3-c70db8a7b969","Type":"ContainerDied","Data":"3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb"}
Jul 20 10:21:07.111084 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.111045    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"
Jul 20 10:21:07.111084 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.111058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5" event={"ID":"75d473c2-c721-4e08-aed3-c70db8a7b969","Type":"ContainerDied","Data":"c42884e3354ddae2df3fd810932f22422b2dcc854d12c7965fb3a4b8d1dd8b79"}
Jul 20 10:21:07.111151 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.111086    2555 scope.go:117] "RemoveContainer" containerID="3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb"
Jul 20 10:21:07.120085 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.120065    2555 scope.go:117] "RemoveContainer" containerID="ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1"
Jul 20 10:21:07.128524 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.128503    2555 scope.go:117] "RemoveContainer" containerID="3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb"
Jul 20 10:21:07.128889 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:21:07.128866    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb\": container with ID starting with 3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb not found: ID does not exist" containerID="3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb"
Jul 20 10:21:07.128948 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.128898    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb"} err="failed to get container status \"3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb\": rpc error: code = NotFound desc = could not find container \"3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb\": container with ID starting with 3aad8ffe89aa428bbdeb5b8eaf12cbcbd22cfd198f9f692029d5558c1d6773cb not found: ID does not exist"
Jul 20 10:21:07.128948 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.128917    2555 scope.go:117] "RemoveContainer" containerID="ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1"
Jul 20 10:21:07.129177 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:21:07.129161    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1\": container with ID starting with ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1 not found: ID does not exist" containerID="ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1"
Jul 20 10:21:07.129234 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.129183    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1"} err="failed to get container status \"ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1\": rpc error: code = NotFound desc = could not find container \"ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1\": container with ID starting with ecb80e3ebdc6b07dc5cfcd89aa4857c144ebe3e160a00c8c5e39049ef23914b1 not found: ID does not exist"
Jul 20 10:21:07.151938 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.151901    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"]
Jul 20 10:21:07.155178 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.155144    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-531344-predictor-684755d79b-ghbr5"]
Jul 20 10:21:07.490901 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:07.490855    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75d473c2-c721-4e08-aed3-c70db8a7b969" path="/var/lib/kubelet/pods/75d473c2-c721-4e08-aed3-c70db8a7b969/volumes"
Jul 20 10:21:10.432589 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.432551    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:10.476746 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.476704    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"]
Jul 20 10:21:10.477156 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.477119    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d473c2-c721-4e08-aed3-c70db8a7b969" containerName="storage-initializer"
Jul 20 10:21:10.477156 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.477139    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d473c2-c721-4e08-aed3-c70db8a7b969" containerName="storage-initializer"
Jul 20 10:21:10.477156 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.477159    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d473c2-c721-4e08-aed3-c70db8a7b969" containerName="storage-initializer"
Jul 20 10:21:10.477459 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.477165    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d473c2-c721-4e08-aed3-c70db8a7b969" containerName="storage-initializer"
Jul 20 10:21:10.477459 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.477240    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d473c2-c721-4e08-aed3-c70db8a7b969" containerName="storage-initializer"
Jul 20 10:21:10.477459 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.477429    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d473c2-c721-4e08-aed3-c70db8a7b969" containerName="storage-initializer"
Jul 20 10:21:10.482001 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.481979    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.485003 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.484976    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:21:10.485003 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.484995    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jbsdb\""
Jul 20 10:21:10.485187 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.484975    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:21:10.486347 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.486323    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:21:10.486443 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.486404    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:21:10.486443 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.486428    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-2ee4d-predictor-serving-cert\""
Jul 20 10:21:10.486652 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.486631    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-2ee4d-kube-rbac-proxy-sar-config\""
Jul 20 10:21:10.492104 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.492075    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"]
Jul 20 10:21:10.593651 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.593605    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-cabundle-cert\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.593651 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.593655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddrw\" (UniqueName: \"kubernetes.io/projected/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-api-access-jddrw\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.593918 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.593692    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.593918 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.593733    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-proxy-tls\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.593918 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.593799    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kserve-provision-location\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.608786 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.608739    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:10.695062 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.694976    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.695062 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.695027    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-proxy-tls\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.695062 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.695057    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kserve-provision-location\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.695351 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.695108    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-cabundle-cert\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.695351 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.695129    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddrw\" (UniqueName: \"kubernetes.io/projected/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-api-access-jddrw\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.695579 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.695557    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kserve-provision-location\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.695878 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.695856    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.695926 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.695856    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-cabundle-cert\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.697598 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.697579    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-proxy-tls\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.704398 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.704371    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddrw\" (UniqueName: \"kubernetes.io/projected/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-api-access-jddrw\") pod \"raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") " pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.793219 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.793177    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:10.954294 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:10.954267    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"]
Jul 20 10:21:10.956595 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:21:10.956565    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b22526_ac03_4063_8fc3_10aeba6ba4b2.slice/crio-fb20b9c67a7e1edc4ddc4ba955ccd432888ec78e6e9f5e87df6ddb243b5ebc19 WatchSource:0}: Error finding container fb20b9c67a7e1edc4ddc4ba955ccd432888ec78e6e9f5e87df6ddb243b5ebc19: Status 404 returned error can't find the container with id fb20b9c67a7e1edc4ddc4ba955ccd432888ec78e6e9f5e87df6ddb243b5ebc19
Jul 20 10:21:11.130299 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:11.130258    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" event={"ID":"c1b22526-ac03-4063-8fc3-10aeba6ba4b2","Type":"ContainerStarted","Data":"6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e"}
Jul 20 10:21:11.130519 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:11.130325    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" event={"ID":"c1b22526-ac03-4063-8fc3-10aeba6ba4b2","Type":"ContainerStarted","Data":"fb20b9c67a7e1edc4ddc4ba955ccd432888ec78e6e9f5e87df6ddb243b5ebc19"}
Jul 20 10:21:15.151543 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:15.151503    2555 generic.go:356] "Generic (PLEG): container finished" podID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerID="6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e" exitCode=0
Jul 20 10:21:15.151959 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:15.151577    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" event={"ID":"c1b22526-ac03-4063-8fc3-10aeba6ba4b2","Type":"ContainerDied","Data":"6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e"}
Jul 20 10:21:16.157956 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:16.157922    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" event={"ID":"c1b22526-ac03-4063-8fc3-10aeba6ba4b2","Type":"ContainerStarted","Data":"0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e"}
Jul 20 10:21:16.158381 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:16.157988    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" event={"ID":"c1b22526-ac03-4063-8fc3-10aeba6ba4b2","Type":"ContainerStarted","Data":"e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539"}
Jul 20 10:21:16.158381 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:16.158273    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:16.158456 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:16.158435    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:16.160054 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:16.160024    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:21:16.179149 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:16.179088    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podStartSLOduration=2.442317038 podStartE2EDuration="6.179071438s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.7367544s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:21:16.177453389 +0000 UTC m=+1449.228382775" watchObservedRunningTime="2026-07-20 10:21:16.179071438 +0000 UTC m=+1449.230000792"
Jul 20 10:21:17.162047 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:17.162006    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:21:22.166917 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:22.166886    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:21:22.167469 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:22.167395    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:21:32.167550 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:32.167503    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:21:42.167983 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:42.167937    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:21:52.167352 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:21:52.167308    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:22:02.168182 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:02.168141    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:22:07.519056 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:07.519028    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:22:07.521502 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:07.521479    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:22:12.167883 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:12.167830    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:22:22.168064 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:22.168022    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:22:32.168403 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:32.168327    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:22:40.589369 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:40.589327    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"]
Jul 20 10:22:40.589865 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:40.589663    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" containerID="cri-o://e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539" gracePeriod=30
Jul 20 10:22:40.589865 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:40.589712    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kube-rbac-proxy" containerID="cri-o://0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e" gracePeriod=30
Jul 20 10:22:41.277412 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:41.277374    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:22:41.550743 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:41.550651    2555 generic.go:356] "Generic (PLEG): container finished" podID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerID="0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e" exitCode=2
Jul 20 10:22:41.550891 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:41.550742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" event={"ID":"c1b22526-ac03-4063-8fc3-10aeba6ba4b2","Type":"ContainerDied","Data":"0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e"}
Jul 20 10:22:42.163067 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:42.163021    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 20 10:22:42.167638 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:42.167603    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:22:45.700685 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.700636    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:22:45.770054 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.770019    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"]
Jul 20 10:22:45.774450 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.774424    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.777208 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.777179    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-35b0f-predictor-serving-cert\""
Jul 20 10:22:45.777378 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.777188    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-35b0f-kube-rbac-proxy-sar-config\""
Jul 20 10:22:45.785422 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.785351    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"]
Jul 20 10:22:45.833501 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.833463    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kserve-provision-location\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.833677 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.833509    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.833677 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.833582    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlj4x\" (UniqueName: \"kubernetes.io/projected/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-api-access-xlj4x\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.833677 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.833606    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-cabundle-cert\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.833677 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.833626    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f170b43a-c316-466d-a90c-1d8c8d9ce04f-proxy-tls\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.844541 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.844514    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:22:45.934469 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934357    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-cabundle-cert\") pod \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") "
Jul 20 10:22:45.934469 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934440    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jddrw\" (UniqueName: \"kubernetes.io/projected/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-api-access-jddrw\") pod \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") "
Jul 20 10:22:45.934737 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934497    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-rbac-proxy-sar-config\") pod \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") "
Jul 20 10:22:45.934737 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934563    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-proxy-tls\") pod \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") "
Jul 20 10:22:45.934737 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934598    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kserve-provision-location\") pod \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\" (UID: \"c1b22526-ac03-4063-8fc3-10aeba6ba4b2\") "
Jul 20 10:22:45.934900 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934758    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kserve-provision-location\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.934900 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934794    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.934900 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934856    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlj4x\" (UniqueName: \"kubernetes.io/projected/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-api-access-xlj4x\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.934900 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934881    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-cabundle-cert\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.935088 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934906    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c1b22526-ac03-4063-8fc3-10aeba6ba4b2" (UID: "c1b22526-ac03-4063-8fc3-10aeba6ba4b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:22:45.935088 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934909    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f170b43a-c316-466d-a90c-1d8c8d9ce04f-proxy-tls\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.935088 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934931    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c1b22526-ac03-4063-8fc3-10aeba6ba4b2" (UID: "c1b22526-ac03-4063-8fc3-10aeba6ba4b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:22:45.935088 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.934968    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1b22526-ac03-4063-8fc3-10aeba6ba4b2" (UID: "c1b22526-ac03-4063-8fc3-10aeba6ba4b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:45.935088 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:22:45.935000    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-serving-cert: secret "raw-sklearn-runtime-35b0f-predictor-serving-cert" not found
Jul 20 10:22:45.935088 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:22:45.935055    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f170b43a-c316-466d-a90c-1d8c8d9ce04f-proxy-tls podName:f170b43a-c316-466d-a90c-1d8c8d9ce04f nodeName:}" failed. No retries permitted until 2026-07-20 10:22:46.435036454 +0000 UTC m=+1539.485965787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f170b43a-c316-466d-a90c-1d8c8d9ce04f-proxy-tls") pod "raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" (UID: "f170b43a-c316-466d-a90c-1d8c8d9ce04f") : secret "raw-sklearn-runtime-35b0f-predictor-serving-cert" not found
Jul 20 10:22:45.935088 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.935074    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:22:45.935088 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.935089    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:22:45.935433 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.935105    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:22:45.935728 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.935704    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kserve-provision-location\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.935728 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.935713    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-cabundle-cert\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.935877 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.935706    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.936898 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.936874    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1b22526-ac03-4063-8fc3-10aeba6ba4b2" (UID: "c1b22526-ac03-4063-8fc3-10aeba6ba4b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:22:45.936995 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.936900    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-api-access-jddrw" (OuterVolumeSpecName: "kube-api-access-jddrw") pod "c1b22526-ac03-4063-8fc3-10aeba6ba4b2" (UID: "c1b22526-ac03-4063-8fc3-10aeba6ba4b2"). InnerVolumeSpecName "kube-api-access-jddrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:22:45.944281 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.944217    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlj4x\" (UniqueName: \"kubernetes.io/projected/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-api-access-xlj4x\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:45.999158 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:45.999118    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:22:46.036489 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.036449    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jddrw\" (UniqueName: \"kubernetes.io/projected/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-kube-api-access-jddrw\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:22:46.036489 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.036483    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1b22526-ac03-4063-8fc3-10aeba6ba4b2-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:22:46.440398 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.440357    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f170b43a-c316-466d-a90c-1d8c8d9ce04f-proxy-tls\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:46.442827 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.442807    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f170b43a-c316-466d-a90c-1d8c8d9ce04f-proxy-tls\") pod \"raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:46.576332 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.576299    2555 generic.go:356] "Generic (PLEG): container finished" podID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerID="e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539" exitCode=0
Jul 20 10:22:46.576531 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.576359    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"
Jul 20 10:22:46.576531 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.576386    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" event={"ID":"c1b22526-ac03-4063-8fc3-10aeba6ba4b2","Type":"ContainerDied","Data":"e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539"}
Jul 20 10:22:46.576531 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.576464    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx" event={"ID":"c1b22526-ac03-4063-8fc3-10aeba6ba4b2","Type":"ContainerDied","Data":"fb20b9c67a7e1edc4ddc4ba955ccd432888ec78e6e9f5e87df6ddb243b5ebc19"}
Jul 20 10:22:46.576531 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.576502    2555 scope.go:117] "RemoveContainer" containerID="0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e"
Jul 20 10:22:46.585016 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.584993    2555 scope.go:117] "RemoveContainer" containerID="e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539"
Jul 20 10:22:46.592557 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.592536    2555 scope.go:117] "RemoveContainer" containerID="6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e"
Jul 20 10:22:46.600958 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.600938    2555 scope.go:117] "RemoveContainer" containerID="0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e"
Jul 20 10:22:46.601239 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:22:46.601219    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e\": container with ID starting with 0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e not found: ID does not exist" containerID="0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e"
Jul 20 10:22:46.601354 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.601325    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e"} err="failed to get container status \"0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e\": rpc error: code = NotFound desc = could not find container \"0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e\": container with ID starting with 0953c711c31e28281096a2075d4523f99a13e4c7f8ae60147e904b33e466301e not found: ID does not exist"
Jul 20 10:22:46.601424 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.601360    2555 scope.go:117] "RemoveContainer" containerID="e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539"
Jul 20 10:22:46.601424 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.601381    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"]
Jul 20 10:22:46.601669 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:22:46.601649    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539\": container with ID starting with e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539 not found: ID does not exist" containerID="e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539"
Jul 20 10:22:46.601710 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.601678    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539"} err="failed to get container status \"e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539\": rpc error: code = NotFound desc = could not find container \"e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539\": container with ID starting with e7be07bc48692f14a3e8d7c8698f3383ad9655a744344b4d4e377d9e1d042539 not found: ID does not exist"
Jul 20 10:22:46.601710 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.601694    2555 scope.go:117] "RemoveContainer" containerID="6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e"
Jul 20 10:22:46.601917 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:22:46.601900    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e\": container with ID starting with 6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e not found: ID does not exist" containerID="6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e"
Jul 20 10:22:46.601955 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.601924    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e"} err="failed to get container status \"6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e\": rpc error: code = NotFound desc = could not find container \"6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e\": container with ID starting with 6b86c697567e41e0d6b62df12c7e023f569531f1d54d738ec3df7a5558a9085e not found: ID does not exist"
Jul 20 10:22:46.608986 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.608960    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2ee4d-predictor-67d5f8f496-wbcqx"]
Jul 20 10:22:46.688228 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.688193    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:46.836929 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:46.836899    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"]
Jul 20 10:22:46.838675 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:22:46.838646    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf170b43a_c316_466d_a90c_1d8c8d9ce04f.slice/crio-113608d4e4d25882bfebeb05b9c9a9066d162a1949a5a43b0272eaf7a182c146 WatchSource:0}: Error finding container 113608d4e4d25882bfebeb05b9c9a9066d162a1949a5a43b0272eaf7a182c146: Status 404 returned error can't find the container with id 113608d4e4d25882bfebeb05b9c9a9066d162a1949a5a43b0272eaf7a182c146
Jul 20 10:22:47.491122 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:47.491084    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" path="/var/lib/kubelet/pods/c1b22526-ac03-4063-8fc3-10aeba6ba4b2/volumes"
Jul 20 10:22:47.581839 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:47.581802    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" event={"ID":"f170b43a-c316-466d-a90c-1d8c8d9ce04f","Type":"ContainerStarted","Data":"3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da"}
Jul 20 10:22:47.582030 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:47.581869    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" event={"ID":"f170b43a-c316-466d-a90c-1d8c8d9ce04f","Type":"ContainerStarted","Data":"113608d4e4d25882bfebeb05b9c9a9066d162a1949a5a43b0272eaf7a182c146"}
Jul 20 10:22:51.602482 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:51.602447    2555 generic.go:356] "Generic (PLEG): container finished" podID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerID="3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da" exitCode=0
Jul 20 10:22:51.602888 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:51.602522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" event={"ID":"f170b43a-c316-466d-a90c-1d8c8d9ce04f","Type":"ContainerDied","Data":"3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da"}
Jul 20 10:22:52.609137 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:52.609100    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" event={"ID":"f170b43a-c316-466d-a90c-1d8c8d9ce04f","Type":"ContainerStarted","Data":"05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985"}
Jul 20 10:22:52.609575 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:52.609166    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" event={"ID":"f170b43a-c316-466d-a90c-1d8c8d9ce04f","Type":"ContainerStarted","Data":"925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190"}
Jul 20 10:22:52.609575 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:52.609376    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:53.613752 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:53.613716    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:53.615129 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:53.615097    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:22:54.618299 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:54.618234    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:22:59.623227 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:59.623197    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:22:59.623754 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:59.623727    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:22:59.656468 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:22:59.656409    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podStartSLOduration=10.844265415 podStartE2EDuration="14.656390694s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.812125279s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:22:52.634894583 +0000 UTC m=+1545.685823939" watchObservedRunningTime="2026-07-20 10:22:59.656390694 +0000 UTC m=+1552.707320049"
Jul 20 10:23:09.624079 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:23:09.624030    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:23:19.623864 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:23:19.623824    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:23:29.623812 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:23:29.623768    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:23:39.623745 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:23:39.623704    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:23:45.107613 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:23:45.107570    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:49.624064 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:23:49.624023    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:23:59.623932 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:23:59.623841    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:24:01.486198 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:01.486155    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:24:11.491810 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:11.491777    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:24:15.909069 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:15.909028    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"]
Jul 20 10:24:15.909660 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:15.909621    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container" containerID="cri-o://925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190" gracePeriod=30
Jul 20 10:24:15.910006 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:15.909977    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kube-rbac-proxy" containerID="cri-o://05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985" gracePeriod=30
Jul 20 10:24:15.929171 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:15.929116    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:16.991140 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:16.991109    2555 generic.go:356] "Generic (PLEG): container finished" podID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerID="05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985" exitCode=2
Jul 20 10:24:16.991562 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:16.991179    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" event={"ID":"f170b43a-c316-466d-a90c-1d8c8d9ce04f","Type":"ContainerDied","Data":"05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985"}
Jul 20 10:24:19.619300 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:19.619229    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 20 10:24:21.161657 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.161622    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:24:21.272239 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.272145    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kserve-provision-location\") pod \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") "
Jul 20 10:24:21.272239 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.272192    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-cabundle-cert\") pod \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") "
Jul 20 10:24:21.272239 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.272213    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-rbac-proxy-sar-config\") pod \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") "
Jul 20 10:24:21.272537 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.272295    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f170b43a-c316-466d-a90c-1d8c8d9ce04f-proxy-tls\") pod \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") "
Jul 20 10:24:21.272537 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.272334    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlj4x\" (UniqueName: \"kubernetes.io/projected/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-api-access-xlj4x\") pod \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\" (UID: \"f170b43a-c316-466d-a90c-1d8c8d9ce04f\") "
Jul 20 10:24:21.272652 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.272589    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f170b43a-c316-466d-a90c-1d8c8d9ce04f" (UID: "f170b43a-c316-466d-a90c-1d8c8d9ce04f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:24:21.272652 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.272626    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f170b43a-c316-466d-a90c-1d8c8d9ce04f" (UID: "f170b43a-c316-466d-a90c-1d8c8d9ce04f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:24:21.272652 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.272641    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f170b43a-c316-466d-a90c-1d8c8d9ce04f" (UID: "f170b43a-c316-466d-a90c-1d8c8d9ce04f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:24:21.274480 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.274446    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-api-access-xlj4x" (OuterVolumeSpecName: "kube-api-access-xlj4x") pod "f170b43a-c316-466d-a90c-1d8c8d9ce04f" (UID: "f170b43a-c316-466d-a90c-1d8c8d9ce04f"). InnerVolumeSpecName "kube-api-access-xlj4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:24:21.274588 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.274503    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f170b43a-c316-466d-a90c-1d8c8d9ce04f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f170b43a-c316-466d-a90c-1d8c8d9ce04f" (UID: "f170b43a-c316-466d-a90c-1d8c8d9ce04f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:24:21.373193 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.373152    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kserve-provision-location\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:24:21.373193 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.373184    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-cabundle-cert\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:24:21.373193 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.373193    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:24:21.373193 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.373202    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f170b43a-c316-466d-a90c-1d8c8d9ce04f-proxy-tls\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:24:21.373540 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.373212    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlj4x\" (UniqueName: \"kubernetes.io/projected/f170b43a-c316-466d-a90c-1d8c8d9ce04f-kube-api-access-xlj4x\") on node \"ip-10-0-129-35.ec2.internal\" DevicePath \"\""
Jul 20 10:24:21.686881 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.686843    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:21.746191 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:21.746157    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:22.014646 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.014559    2555 generic.go:356] "Generic (PLEG): container finished" podID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerID="925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190" exitCode=0
Jul 20 10:24:22.014646 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.014633    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"
Jul 20 10:24:22.014833 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.014646    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" event={"ID":"f170b43a-c316-466d-a90c-1d8c8d9ce04f","Type":"ContainerDied","Data":"925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190"}
Jul 20 10:24:22.014833 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.014751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq" event={"ID":"f170b43a-c316-466d-a90c-1d8c8d9ce04f","Type":"ContainerDied","Data":"113608d4e4d25882bfebeb05b9c9a9066d162a1949a5a43b0272eaf7a182c146"}
Jul 20 10:24:22.014833 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.014792    2555 scope.go:117] "RemoveContainer" containerID="05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985"
Jul 20 10:24:22.023185 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.023163    2555 scope.go:117] "RemoveContainer" containerID="925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190"
Jul 20 10:24:22.031073 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.030880    2555 scope.go:117] "RemoveContainer" containerID="3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da"
Jul 20 10:24:22.034400 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.034376    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"]
Jul 20 10:24:22.039669 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.039649    2555 scope.go:117] "RemoveContainer" containerID="05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985"
Jul 20 10:24:22.039965 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:24:22.039945    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985\": container with ID starting with 05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985 not found: ID does not exist" containerID="05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985"
Jul 20 10:24:22.040023 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.039978    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985"} err="failed to get container status \"05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985\": rpc error: code = NotFound desc = could not find container \"05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985\": container with ID starting with 05e3b4332d33e5ed5209ff08533567c29d6324b51f43d0f807df2480573be985 not found: ID does not exist"
Jul 20 10:24:22.040023 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.039997    2555 scope.go:117] "RemoveContainer" containerID="925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190"
Jul 20 10:24:22.040331 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:24:22.040309    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190\": container with ID starting with 925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190 not found: ID does not exist" containerID="925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190"
Jul 20 10:24:22.040410 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.040338    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190"} err="failed to get container status \"925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190\": rpc error: code = NotFound desc = could not find container \"925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190\": container with ID starting with 925c3679b16e0755508b44f7c02250d7ee8a654e1df483e3dda821f58fd6e190 not found: ID does not exist"
Jul 20 10:24:22.040410 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.040354    2555 scope.go:117] "RemoveContainer" containerID="3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da"
Jul 20 10:24:22.040632 ip-10-0-129-35 kubenswrapper[2555]: E0720 10:24:22.040615    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da\": container with ID starting with 3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da not found: ID does not exist" containerID="3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da"
Jul 20 10:24:22.040681 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.040640    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da"} err="failed to get container status \"3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da\": rpc error: code = NotFound desc = could not find container \"3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da\": container with ID starting with 3754356f3e2b8ccbf228ed3644ff653230b197ce9c3e05cfb8e7a408527c98da not found: ID does not exist"
Jul 20 10:24:22.040754 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:22.040736    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-35b0f-predictor-84b5c8555b-648dq"]
Jul 20 10:24:23.492008 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:23.491974    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" path="/var/lib/kubelet/pods/f170b43a-c316-466d-a90c-1d8c8d9ce04f/volumes"
Jul 20 10:24:28.509868 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:28.509827    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:50.641159 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.641121    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:50.795068 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795031    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-srqvl/must-gather-flk9n"]
Jul 20 10:24:50.795429 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795414    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="storage-initializer"
Jul 20 10:24:50.795492 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795432    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="storage-initializer"
Jul 20 10:24:50.795492 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795443    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container"
Jul 20 10:24:50.795492 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795449    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container"
Jul 20 10:24:50.795492 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795458    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kube-rbac-proxy"
Jul 20 10:24:50.795492 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795463    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kube-rbac-proxy"
Jul 20 10:24:50.795492 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795471    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="storage-initializer"
Jul 20 10:24:50.795492 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795476    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="storage-initializer"
Jul 20 10:24:50.795806 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795484    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container"
Jul 20 10:24:50.795806 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795520    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container"
Jul 20 10:24:50.795806 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795548    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kube-rbac-proxy"
Jul 20 10:24:50.795806 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795554    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kube-rbac-proxy"
Jul 20 10:24:50.795806 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795620    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kserve-container"
Jul 20 10:24:50.795806 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795629    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kserve-container"
Jul 20 10:24:50.795806 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795636    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b22526-ac03-4063-8fc3-10aeba6ba4b2" containerName="kube-rbac-proxy"
Jul 20 10:24:50.795806 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.795652    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f170b43a-c316-466d-a90c-1d8c8d9ce04f" containerName="kube-rbac-proxy"
Jul 20 10:24:50.798702 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.798686    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srqvl/must-gather-flk9n"
Jul 20 10:24:50.800856 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.800820    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:50.801526 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.801505    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-srqvl\"/\"openshift-service-ca.crt\""
Jul 20 10:24:50.801653 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.801525    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-srqvl\"/\"kube-root-ca.crt\""
Jul 20 10:24:50.802642 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.802627    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-srqvl\"/\"default-dockercfg-wnb8s\""
Jul 20 10:24:50.807416 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.807396    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srqvl/must-gather-flk9n"]
Jul 20 10:24:50.855051 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.855011    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:50.935532 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.935441    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh556\" (UniqueName: \"kubernetes.io/projected/f39d9956-c07f-4a2d-b288-482f46018464-kube-api-access-nh556\") pod \"must-gather-flk9n\" (UID: \"f39d9956-c07f-4a2d-b288-482f46018464\") " pod="openshift-must-gather-srqvl/must-gather-flk9n"
Jul 20 10:24:50.935532 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:50.935503    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f39d9956-c07f-4a2d-b288-482f46018464-must-gather-output\") pod \"must-gather-flk9n\" (UID: \"f39d9956-c07f-4a2d-b288-482f46018464\") " pod="openshift-must-gather-srqvl/must-gather-flk9n"
Jul 20 10:24:51.036881 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:51.036840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f39d9956-c07f-4a2d-b288-482f46018464-must-gather-output\") pod \"must-gather-flk9n\" (UID: \"f39d9956-c07f-4a2d-b288-482f46018464\") " pod="openshift-must-gather-srqvl/must-gather-flk9n"
Jul 20 10:24:51.037066 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:51.036980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh556\" (UniqueName: \"kubernetes.io/projected/f39d9956-c07f-4a2d-b288-482f46018464-kube-api-access-nh556\") pod \"must-gather-flk9n\" (UID: \"f39d9956-c07f-4a2d-b288-482f46018464\") " pod="openshift-must-gather-srqvl/must-gather-flk9n"
Jul 20 10:24:51.037214 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:51.037193    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f39d9956-c07f-4a2d-b288-482f46018464-must-gather-output\") pod \"must-gather-flk9n\" (UID: \"f39d9956-c07f-4a2d-b288-482f46018464\") " pod="openshift-must-gather-srqvl/must-gather-flk9n"
Jul 20 10:24:51.046846 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:51.046820    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh556\" (UniqueName: \"kubernetes.io/projected/f39d9956-c07f-4a2d-b288-482f46018464-kube-api-access-nh556\") pod \"must-gather-flk9n\" (UID: \"f39d9956-c07f-4a2d-b288-482f46018464\") " pod="openshift-must-gather-srqvl/must-gather-flk9n"
Jul 20 10:24:51.121106 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:51.121064    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srqvl/must-gather-flk9n"
Jul 20 10:24:51.257022 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:51.256990    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srqvl/must-gather-flk9n"]
Jul 20 10:24:51.259781 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:24:51.259747    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf39d9956_c07f_4a2d_b288_482f46018464.slice/crio-6db07a661e5c05f7f45c186939f7e06ee788b9711a01b0ca12a38f068fa73bbe WatchSource:0}: Error finding container 6db07a661e5c05f7f45c186939f7e06ee788b9711a01b0ca12a38f068fa73bbe: Status 404 returned error can't find the container with id 6db07a661e5c05f7f45c186939f7e06ee788b9711a01b0ca12a38f068fa73bbe
Jul 20 10:24:51.261730 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:51.261708    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:24:52.147942 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:52.147908    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srqvl/must-gather-flk9n" event={"ID":"f39d9956-c07f-4a2d-b288-482f46018464","Type":"ContainerStarted","Data":"6db07a661e5c05f7f45c186939f7e06ee788b9711a01b0ca12a38f068fa73bbe"}
Jul 20 10:24:53.157807 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:53.157755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srqvl/must-gather-flk9n" event={"ID":"f39d9956-c07f-4a2d-b288-482f46018464","Type":"ContainerStarted","Data":"d8c9e1564197f265e4af1426282058e65a58483115e0019120138e52e8d60814"}
Jul 20 10:24:53.158418 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:53.158376    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srqvl/must-gather-flk9n" event={"ID":"f39d9956-c07f-4a2d-b288-482f46018464","Type":"ContainerStarted","Data":"84d67ab4e16f7525d016adc49955c6b48f91b35aaf1faf9fb6b3561276316b4c"}
Jul 20 10:24:53.573883 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:53.573788    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5r7zg_3796601d-5123-4213-8c2d-453d6212d472/global-pull-secret-syncer/0.log"
Jul 20 10:24:53.743567 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:53.743533    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log"
Jul 20 10:24:53.878085 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:53.878053    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-35.ec2.internal_ba2adbd0e228b3393b7ea7327f58790b/haproxy/0.log"
Jul 20 10:24:56.205476 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:56.205428    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:56.305457 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:56.305417    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2gwd_d8bee537-c2d5-4977-a3fd-b8d6cc021900/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:58.326449 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.326352    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-z5mbc_58f197ee-2586-441b-b591-66090ab5dd32/cluster-monitoring-operator/0.log"
Jul 20 10:24:58.355953 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.355913    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-njr9r_743df18d-8914-47c2-b93d-2245a9bd400f/kube-state-metrics/0.log"
Jul 20 10:24:58.387387 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.387352    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-njr9r_743df18d-8914-47c2-b93d-2245a9bd400f/kube-rbac-proxy-main/0.log"
Jul 20 10:24:58.414363 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.414331    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-njr9r_743df18d-8914-47c2-b93d-2245a9bd400f/kube-rbac-proxy-self/0.log"
Jul 20 10:24:58.477856 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.477821    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-m7fmr_080a463c-1129-46fd-99ac-e2c1505234a9/monitoring-plugin/0.log"
Jul 20 10:24:58.608785 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.608752    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lltk8_2c315b98-9976-4328-bad4-d33913f17621/node-exporter/0.log"
Jul 20 10:24:58.636316 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.636286    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lltk8_2c315b98-9976-4328-bad4-d33913f17621/kube-rbac-proxy/0.log"
Jul 20 10:24:58.663322 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.663293    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lltk8_2c315b98-9976-4328-bad4-d33913f17621/init-textfile/0.log"
Jul 20 10:24:58.770998 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.770965    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-kqrjp_1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1/kube-rbac-proxy-main/0.log"
Jul 20 10:24:58.791871 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.791843    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-kqrjp_1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1/kube-rbac-proxy-self/0.log"
Jul 20 10:24:58.813665 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:58.813616    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-kqrjp_1ad22f6a-96b8-4ffc-9cf5-cb85ea6ff5b1/openshift-state-metrics/0.log"
Jul 20 10:24:59.087717 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:59.087607    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-747bc6f6bd-cqkqd_8f73a5b8-6d5a-4708-94d3-1c826e6894d2/telemeter-client/0.log"
Jul 20 10:24:59.110754 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:59.110726    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-747bc6f6bd-cqkqd_8f73a5b8-6d5a-4708-94d3-1c826e6894d2/reload/0.log"
Jul 20 10:24:59.131515 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:59.131486    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-747bc6f6bd-cqkqd_8f73a5b8-6d5a-4708-94d3-1c826e6894d2/kube-rbac-proxy/0.log"
Jul 20 10:24:59.163391 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:59.163363    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd6b67cdf-5wcd5_e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd/thanos-query/0.log"
Jul 20 10:24:59.183874 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:59.183841    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd6b67cdf-5wcd5_e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd/kube-rbac-proxy-web/0.log"
Jul 20 10:24:59.207728 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:59.207694    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd6b67cdf-5wcd5_e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd/kube-rbac-proxy/0.log"
Jul 20 10:24:59.231417 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:59.231376    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd6b67cdf-5wcd5_e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd/prom-label-proxy/0.log"
Jul 20 10:24:59.253267 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:59.253223    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd6b67cdf-5wcd5_e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd/kube-rbac-proxy-rules/0.log"
Jul 20 10:24:59.276787 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:24:59.276758    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd6b67cdf-5wcd5_e8eae26e-a89d-4b02-8ca0-cb4fc1ac19bd/kube-rbac-proxy-metrics/0.log"
Jul 20 10:25:01.447460 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:01.447432    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-595fc7c78d-qpn9d_7944cd0f-9a9f-4449-85a3-983ea9036841/console/0.log"
Jul 20 10:25:01.481300 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:01.481271    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-6pxc9_00319ed1-836b-4b3a-a2e8-da8d38bb3a66/download-server/0.log"
Jul 20 10:25:02.409701 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.409629    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-srqvl/must-gather-flk9n" podStartSLOduration=11.479835223 podStartE2EDuration="12.409603352s" totalImagesPullingTime="929.768129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:24:53.180197293 +0000 UTC m=+1666.231126649" watchObservedRunningTime="2026-07-20 10:25:02.409603352 +0000 UTC m=+1675.460532708"
Jul 20 10:25:02.410301 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.410267    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"]
Jul 20 10:25:02.416500 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.416468    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.425105 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.425070    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"]
Jul 20 10:25:02.453821 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.453780    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-podres\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.454429 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.454402    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-proc\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.454593 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.454577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r97n8\" (UniqueName: \"kubernetes.io/projected/639e505d-d1b8-46e1-9789-d90f10571c49-kube-api-access-r97n8\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.454746 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.454730    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-sys\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.454850 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.454838    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-lib-modules\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.555807 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.555763    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-podres\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.556078 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.556030    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-podres\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.556295 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.556091    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-proc\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.556513 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.556495    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r97n8\" (UniqueName: \"kubernetes.io/projected/639e505d-d1b8-46e1-9789-d90f10571c49-kube-api-access-r97n8\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.556714 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.556697    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-sys\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.556868 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.556852    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-lib-modules\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.556992 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.556964    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-sys\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.557111 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.556172    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-proc\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.557111 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.557017    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/639e505d-d1b8-46e1-9789-d90f10571c49-lib-modules\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.567271 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.567214    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r97n8\" (UniqueName: \"kubernetes.io/projected/639e505d-d1b8-46e1-9789-d90f10571c49-kube-api-access-r97n8\") pod \"perf-node-gather-daemonset-rlxz9\" (UID: \"639e505d-d1b8-46e1-9789-d90f10571c49\") " pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.731061 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.730972    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:02.913798 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:02.913764    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"]
Jul 20 10:25:02.916652 ip-10-0-129-35 kubenswrapper[2555]: W0720 10:25:02.916618    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod639e505d_d1b8_46e1_9789_d90f10571c49.slice/crio-4fa39f719a5fb84efef51f978584d1f5ef0998ecdc2fdbb37dc089157634faaa WatchSource:0}: Error finding container 4fa39f719a5fb84efef51f978584d1f5ef0998ecdc2fdbb37dc089157634faaa: Status 404 returned error can't find the container with id 4fa39f719a5fb84efef51f978584d1f5ef0998ecdc2fdbb37dc089157634faaa
Jul 20 10:25:03.032174 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:03.032091    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mlj89_c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3/dns/0.log"
Jul 20 10:25:03.052453 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:03.052426    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mlj89_c5d0e67f-d74d-4e9b-8769-fdc9e2b56fa3/kube-rbac-proxy/0.log"
Jul 20 10:25:03.120441 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:03.120344    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tfh95_a0274aa1-d4b8-463c-8f61-d888fe04e631/dns-node-resolver/0.log"
Jul 20 10:25:03.224434 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:03.224392    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9" event={"ID":"639e505d-d1b8-46e1-9789-d90f10571c49","Type":"ContainerStarted","Data":"513e82fa5fa83876ffc629805c5834c9d2fb5dd653b77a57afbd7735aabb0f6a"}
Jul 20 10:25:03.224625 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:03.224458    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9" event={"ID":"639e505d-d1b8-46e1-9789-d90f10571c49","Type":"ContainerStarted","Data":"4fa39f719a5fb84efef51f978584d1f5ef0998ecdc2fdbb37dc089157634faaa"}
Jul 20 10:25:03.224625 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:03.224548    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:03.242186 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:03.242128    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9" podStartSLOduration=1.242109757 podStartE2EDuration="1.242109757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:25:03.241430353 +0000 UTC m=+1676.292359709" watchObservedRunningTime="2026-07-20 10:25:03.242109757 +0000 UTC m=+1676.293039113"
Jul 20 10:25:03.587079 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:03.587046    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-687f897475-jbl2v_e7e64ff1-09f6-42f4-9fc2-1ab9ac94f7f2/registry/0.log"
Jul 20 10:25:03.606406 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:03.606372    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cm5ps_58dc4f4e-0a78-4b1d-af07-30685a4498d0/node-ca/0.log"
Jul 20 10:25:04.371338 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:04.371312    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67c67595d6-ltx5q_17037412-22f7-435b-9b7c-54d02129d8c5/router/0.log"
Jul 20 10:25:04.714921 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:04.714839    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jgts6_630a07a2-1674-43f8-82d6-08df700b040c/serve-healthcheck-canary/0.log"
Jul 20 10:25:05.108369 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:05.108344    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8x59k_903c240d-9f9b-41d5-a837-95aa6feff967/kube-rbac-proxy/0.log"
Jul 20 10:25:05.127524 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:05.127497    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8x59k_903c240d-9f9b-41d5-a837-95aa6feff967/exporter/0.log"
Jul 20 10:25:05.149708 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:05.149668    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8x59k_903c240d-9f9b-41d5-a837-95aa6feff967/extractor/0.log"
Jul 20 10:25:07.554383 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:07.554348    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-wpj4f_cf9c9e27-5b07-455e-bc14-9d514d052725/server/0.log"
Jul 20 10:25:07.709443 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:07.709403    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-cr6cj_ef21c342-5785-4b06-84cb-89a99bc7083f/manager/0.log"
Jul 20 10:25:07.826044 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:07.825961    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-v4nj6_81cb20cb-8c0d-45a3-a410-2afb50bdbe8e/seaweedfs/0.log"
Jul 20 10:25:09.243348 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:09.243315    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-srqvl/perf-node-gather-daemonset-rlxz9"
Jul 20 10:25:13.542602 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:13.542571    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68fww_bfcd6077-e7cf-4e17-829b-514131357a1a/kube-multus-additional-cni-plugins/0.log"
Jul 20 10:25:13.566354 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:13.566327    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68fww_bfcd6077-e7cf-4e17-829b-514131357a1a/egress-router-binary-copy/0.log"
Jul 20 10:25:13.588408 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:13.588369    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68fww_bfcd6077-e7cf-4e17-829b-514131357a1a/cni-plugins/0.log"
Jul 20 10:25:13.608827 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:13.608794    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68fww_bfcd6077-e7cf-4e17-829b-514131357a1a/bond-cni-plugin/0.log"
Jul 20 10:25:13.630350 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:13.630312    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68fww_bfcd6077-e7cf-4e17-829b-514131357a1a/routeoverride-cni/0.log"
Jul 20 10:25:13.651361 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:13.651334    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68fww_bfcd6077-e7cf-4e17-829b-514131357a1a/whereabouts-cni-bincopy/0.log"
Jul 20 10:25:13.671095 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:13.671066    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68fww_bfcd6077-e7cf-4e17-829b-514131357a1a/whereabouts-cni/0.log"
Jul 20 10:25:13.870273 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:13.870220    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cnzgt_2db3a8a2-e368-4a2a-aae2-4ed24d9e4eac/kube-multus/0.log"
Jul 20 10:25:14.006864 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:14.006827    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p24w6_ed36d59b-3fc9-43ea-9506-f28d6955b7a3/network-metrics-daemon/0.log"
Jul 20 10:25:14.024938 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:14.024895    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p24w6_ed36d59b-3fc9-43ea-9506-f28d6955b7a3/kube-rbac-proxy/0.log"
Jul 20 10:25:14.938282 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:14.938233    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-controller/0.log"
Jul 20 10:25:14.954681 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:14.954645    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/1.log"
Jul 20 10:25:14.954681 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:14.954663    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovn-acl-logging/0.log"
Jul 20 10:25:14.972362 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:14.972327    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/kube-rbac-proxy-node/0.log"
Jul 20 10:25:14.992122 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:14.992093    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/kube-rbac-proxy-ovn-metrics/0.log"
Jul 20 10:25:15.008718 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:15.008686    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/northd/0.log"
Jul 20 10:25:15.029542 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:15.029518    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/nbdb/0.log"
Jul 20 10:25:15.051555 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:15.051516    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/sbdb/0.log"
Jul 20 10:25:15.116876 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:15.116842    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dsbz_2851d0a8-1480-444d-bc7e-ab21239553df/ovnkube-controller/0.log"
Jul 20 10:25:16.333541 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:16.333510    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-q7zvj_1d7b7476-9151-41f4-a037-54a40c87cdc4/network-check-target-container/0.log"
Jul 20 10:25:17.252618 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:17.252586    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-kpskk_b75520ae-fef6-42fe-a019-59a2f6b04a24/iptables-alerter/0.log"
Jul 20 10:25:17.982212 ip-10-0-129-35 kubenswrapper[2555]: I0720 10:25:17.982179    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xzt4t_cba89306-92d2-4c9f-a577-d992c350a925/tuned/0.log"