Jul 16 17:31:33.603480 ip-10-0-132-218 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 17:31:33.603491 ip-10-0-132-218 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 17:31:33.603498 ip-10-0-132-218 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 17:31:33.603712 ip-10-0-132-218 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 17:31:43.741688 ip-10-0-132-218 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 17:31:43.741704 ip-10-0-132-218 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 215bbb3d5d884e509e129fae70f196e3 --
Jul 16 17:33:42.895864 ip-10-0-132-218 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.368401    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373376    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373532    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373538    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373542    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373545    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373569    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373572    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:33:43.433663 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373575    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373578    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373581    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373583    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373586    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373589    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373591    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373595    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373598    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373601    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373603    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373606    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373609    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373612    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373615    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373617    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373620    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373623    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373627    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373630    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:33:43.434517 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373633    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373636    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373638    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373643    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373648    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373652    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373656    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373659    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373662    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373664    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373669    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373673    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373676    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373679    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373682    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373685    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373688    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373692    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373695    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:33:43.435044 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373698    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373702    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373705    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373708    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373710    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373713    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373716    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373719    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373721    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373724    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373727    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373730    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373733    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373736    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373739    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373742    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373744    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373747    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373749    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373752    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:33:43.435593 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373755    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373759    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373761    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373765    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373768    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373770    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373773    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373775    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373778    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373781    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373783    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373786    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373790    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373793    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373796    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373799    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373802    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373805    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373808    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373810    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:33:43.436213 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373813    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373817    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373820    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373823    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373826    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373828    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373831    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373833    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373836    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373838    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373841    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373844    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.373847    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374633    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374642    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374646    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374649    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374653    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374656    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374659    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:33:43.436771 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374662    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374664    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374667    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374670    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374673    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374675    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374678    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374680    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374683    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374685    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374688    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374691    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374694    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374696    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374698    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374701    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374704    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374708    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374711    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374714    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:33:43.437337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374716    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374719    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374722    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374725    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374728    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374731    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374733    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374736    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374739    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374741    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374744    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374746    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374749    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374752    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374755    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374757    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374761    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374764    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374766    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:33:43.437935 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374769    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374772    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374774    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374777    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374780    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374783    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374785    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374788    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374790    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374793    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374796    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374798    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374801    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374803    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374806    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374809    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374811    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374813    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374816    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374819    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:33:43.438496 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374821    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374823    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374826    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374829    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374831    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374833    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374840    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374842    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374845    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374848    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374851    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374853    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374856    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374859    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374864    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374867    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374870    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374874    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374877    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374880    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:33:43.439090 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374883    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374886    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374889    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374892    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374894    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374897    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374900    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374902    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374907    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374910    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374913    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374916    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.374918    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375336    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375341    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375344    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375347    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375350    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375355    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:33:43.439744 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375358    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375361    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375365    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375369    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375372    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375375    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375378    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375381    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375384    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375387    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375390    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375393    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375396    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375399    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375402    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375404    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375407    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375410    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375412    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:33:43.440271 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375415    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375417    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375420    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375422    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375425    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375429    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375432    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375435    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375438    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375441    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375444    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375447    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375450    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375453    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375456    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375458    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375461    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375465    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375468    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:33:43.440852 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375471    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375474    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375476    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375479    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375481    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375485    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375487    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375490    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375492    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375495    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375498    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375500    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375503    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375505    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375508    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375511    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375513    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375517    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375519    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375522    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:33:43.441515 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375524    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375527    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375529    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375532    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375534    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375539    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375542    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375567    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375570    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375572    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375575    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375577    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375580    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375582    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375585    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375587    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375590    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375593    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375596    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375598    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:33:43.442195 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375601    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375603    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375606    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375608    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375611    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375614    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375616    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375619    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375621    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375625    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375628    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375630    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375633    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375635    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.375638    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375707    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375714    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375721    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375726    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375730    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375733    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 17:33:43.442998 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375737    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375742    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375745    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375748    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375752    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375755    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375758    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375761    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375764    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375767    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375770    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375773    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375778    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375780    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375784    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375787    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375790    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375794    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375797    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375800    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375804    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375807    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375811    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375814    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 17:33:43.443847 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375817    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375820    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375824    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375827    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375831    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375834    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375837    2563 flags.go:64] FLAG: --enable-server="true"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375840    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375845    2563 flags.go:64] FLAG: --event-burst="100"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375848    2563 flags.go:64] FLAG: --event-qps="50"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375851    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375854    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375857    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375866    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375869    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375872    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375875    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375878    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375881    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375885    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375888    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375891    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375894    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375896    2563 flags.go:64] FLAG: --feature-gates=""
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375900    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 17:33:43.444916 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375903    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375906    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375909    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375913    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375916    2563 flags.go:64] FLAG: --help="false"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375919    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375922    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375925    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375928    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375931    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375935    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375939    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375942    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375945    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375947    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375950    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375953    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375956    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375959    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375961    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375964    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375967    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375970    2563 flags.go:64] FLAG: --lock-file=""
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375973    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 17:33:43.521807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375976    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 17:33:43.460957 ip-10-0-132-218 systemd[1]: Started Kubernetes Kubelet.
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375979    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375984    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375988    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375991    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375993    2563 flags.go:64] FLAG: --logging-format="text"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375996    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.375999    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376002    2563 flags.go:64] FLAG: --manifest-url=""
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376005    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376009    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376013    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376030    2563 flags.go:64] FLAG: --max-pods="110"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376034    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376038    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376041    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376044    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376048    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376051    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376055    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376063    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376066    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376069    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 17:33:43.523069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376073    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376076    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376081    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376084    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376087    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376090    2563 flags.go:64] FLAG: --port="10250"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376093    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376096    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0338d040c19e5b71f"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376099    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376102    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376105    2563 flags.go:64] FLAG: --register-node="true"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376108    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376111    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376115    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376118    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376121    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376124    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376127    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376130    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376132    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376137    2563 flags.go:64] FLAG: --runonce="false"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376140    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376143    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376146    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376148    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 17:33:43.523923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376151    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376154    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376157    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376160    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376164    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376167    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376170    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376173    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376176    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376179    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376182    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376187    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376190    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376193    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376198    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376200    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376203    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376206    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376209    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376212    2563 flags.go:64] FLAG: --v="2"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376216    2563 flags.go:64] FLAG: --version="false"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376221    2563 flags.go:64] FLAG: --vmodule=""
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376224    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.376228    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376345    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:33:43.524673 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376349    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376352    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376357    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376360    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376363    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376366    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376368    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376371    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376373    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376376    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376379    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376382    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376385    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376389    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376393    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376395    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376398    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376401    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376403    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376406    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:33:43.525372 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376408    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376411    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376414    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376416    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376419    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376421    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376424    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376426    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376429    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376432    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376434    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376437    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376439    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376442    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376446    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376449    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376451    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376453    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376456    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376458    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:33:43.526003 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376461    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376463    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376466    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376470    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376473    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376475    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376478    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376480    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376483    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376485    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376488    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376490    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376495    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376498    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376501    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376505    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376508    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376511    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376513    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:33:43.526640 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376516    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376519    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376522    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376524    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376527    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376530    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376533    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376537    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376539    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376542    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376545    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376563    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376566    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376569    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376572    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376574    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376578    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376581    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376583    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:33:43.527272 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376586    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376589    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376592    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376594    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376597    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376599    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376602    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376605    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376608    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376610    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376613    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376616    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376618    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376621    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376623    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376626    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376628    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376635    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376638    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:33:43.528255 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.376640    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.377329    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.384466    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.384483    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.384512    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.384521    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384606    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384613    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384618    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384622    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384625    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384628    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384631    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384633    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:33:43.528837 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384637    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384640    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384642    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384645    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384648    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384650    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384653    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384656    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384658    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384661    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384664    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384666    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384669    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384671    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384674    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384677    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384681    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384683    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384686    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384688    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:33:43.529794 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384691    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384694    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384697    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384699    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384702    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384705    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384707    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384710    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384712    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384714    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384717    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384719    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384722    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384725    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384728    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384730    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384734    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384737    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384740    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:33:43.530359 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384743    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384746    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384748    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384751    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384754    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384757    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384759    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384762    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384765    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384767    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384771    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384774    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384777    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384779    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384782    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384785    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384787    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384790    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384793    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:33:43.531291 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384797    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384800    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384803    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384805    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384808    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384810    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384813    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384815    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384818    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384821    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384823    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384826    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384829    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384831    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384835    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384837    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384840    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384842    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384845    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384847    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:33:43.531979 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384854    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384857    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384859    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384861    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384865    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384869    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384872    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384874    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384877    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384880    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384882    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384885    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.384887    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.384892    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385005    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:33:43.532667 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385010    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385012    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385015    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385018    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385020    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385023    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385025    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385028    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385030    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385033    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385035    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385038    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385040    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385043    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385045    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385047    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385050    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385054    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385057    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385059    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:33:43.533242 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385062    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385066    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385073    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385078    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385080    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385083    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385085    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385088    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385091    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385093    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385096    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385098    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385101    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385103    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385105    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385108    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385111    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385113    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385116    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385118    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:33:43.534013 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385121    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385123    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385126    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385129    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385132    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385134    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385137    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385139    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385142    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385146    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385148    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385151    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385154    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385156    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385160    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385165    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385168    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385171    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385174    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:33:43.534525 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385177    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385179    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385182    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385185    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385187    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385189    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385192    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385195    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385197    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385200    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385203    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385205    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385208    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385210    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385213    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385215    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385218    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385220    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385223    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385225    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:33:43.535008 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385228    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385230    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385234    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385238    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385241    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385244    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385247    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385250    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385253    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385256    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385259    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385261    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385264    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385266    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385269    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385272    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385274    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385277    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:33:43.535509 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:43.385279    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.385285    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.385410    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.387475    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.388662    2563 server.go:1019] "Starting client certificate rotation"
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.388759    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.388798    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.390990    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.392140    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.407965    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.411498    2563 log.go:25] "Validated CRI v1 image API"
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.418097    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 17:33:43.535963 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.420092    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 17:33:43.536298 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.424853    2563 fs.go:135] Filesystem UUIDs: map[1f02096b-207b-43c3-8af8-66e9ff0ee071:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9bc28602-b2a1-44ed-9d37-a6fe4bd52c70:/dev/nvme0n1p3]
Jul 16 17:33:43.536298 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.424869    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.431341    2563 manager.go:216] Machine: {Timestamp:2026-07-16 17:33:43.428979271 +0000 UTC m=+0.415514870 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099950 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b812a79e71de2b0307775f3630b64 SystemUUID:ec2b812a-79e7-1de2-b030-7775f3630b64 BootID:215bbb3d-5d88-4e50-9e12-9fae70f196e3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:de:8e:c5:55:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:de:8e:c5:55:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:b9:1b:5c:42:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.431460    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.431619    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.434022    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.434054    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-218.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.434721    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.434732    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.434745    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.435315    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.436677    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.440898    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.441024    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.441058    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.441069    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.441086    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.441095    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.441108    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 17:33:43.536729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.441477    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.441506    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.452281    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.453770    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.454669    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.454705    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.454893    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.454909    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.454917    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.454925    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.454936    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.454944    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.455318    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.455327    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.455338    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.455347    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.455366    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.455377    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.456320    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.456330    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.459178    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h2wlc"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.460045    2563 server.go:1268] "Started kubelet"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.460134    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.460130    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.460259    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.461433    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.462119    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.464184    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.464343    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-218.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.467977    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h2wlc"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.468989    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.469009    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.469103    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.469544    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.469569    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.469584    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.469672    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.469679    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.475666    2563 factory.go:55] Registering systemd factory
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.475679    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.475750    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.475800    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.476043    2563 factory.go:153] Registering CRI-O factory
Jul 16 17:33:43.537218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.476059    2563 factory.go:223] Registration of the crio container factory successfully
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.476107    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.476128    2563 factory.go:103] Registering Raw factory
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.476152    2563 manager.go:1196] Started watching for new ooms in manager
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.476362    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.476971    2563 manager.go:349] Starting recovery of all containers
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.475658    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-218.ec2.internal.18c2d5918a56202c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-218.ec2.internal,UID:ip-10-0-132-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-218.ec2.internal,},FirstTimestamp:2026-07-16 17:33:43.460012076 +0000 UTC m=+0.446547712,LastTimestamp:2026-07-16 17:33:43.460012076 +0000 UTC m=+0.446547712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-218.ec2.internal,}"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.487702    2563 manager.go:354] Recovery completed
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.492808    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.494438    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.494472    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.494483    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.495020    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.495032    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.495021    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.495069    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.495963    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.495991    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.496006    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.496935    2563 policy_none.go:49] "None policy: Start"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.496954    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.496963    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.498943    2563 policy_none.go:47] "Start"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.532261    2563 manager.go:337] "Starting Device Plugin manager"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.532438    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.532451    2563 server.go:87] "Starting device plugin registration server"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.532718    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.532730    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.532819    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.532912    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.532918    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.533353    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 17:33:43.538290 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.533389    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:43.650466 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.632815    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:33:43.650466 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.633800    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:33:43.650466 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.633831    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:33:43.650466 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.633841    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:33:43.650466 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.633865    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.650466 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.646331    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.650466 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.646355    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-218.ec2.internal\": node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:43.730196 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.666312    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 17:33:43.730196 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.667503    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 17:33:43.730196 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.667546    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 17:33:43.730196 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.667582    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 17:33:43.730196 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.667617    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 17:33:43.730196 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.689643    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:33:43.730196 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.718899    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.767788    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal"]
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.767875    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.768857    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.768887    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.768899    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.770753    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.770892    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.770917    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.771108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b7aea08f316a1dd14e58ba5cd1a755a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-218.ec2.internal\" (UID: \"2b7aea08f316a1dd14e58ba5cd1a755a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.777309    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.777333    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.777344    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.777311    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.777395    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.777405    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.779223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.779245    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.780462    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.780483    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:33:43.819025 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.780491    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:33:43.821200 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.819472    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:43.884416 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.872200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b7aea08f316a1dd14e58ba5cd1a755a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-218.ec2.internal\" (UID: \"2b7aea08f316a1dd14e58ba5cd1a755a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.884416 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.872239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9544c8d8a199e4cc63eaab616f80015e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal\" (UID: \"9544c8d8a199e4cc63eaab616f80015e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.884416 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.872261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9544c8d8a199e4cc63eaab616f80015e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal\" (UID: \"9544c8d8a199e4cc63eaab616f80015e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.884416 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.872297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b7aea08f316a1dd14e58ba5cd1a755a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-218.ec2.internal\" (UID: \"2b7aea08f316a1dd14e58ba5cd1a755a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.941890 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.920447    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:43.941890 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.924338    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-218.ec2.internal\" not found" node="ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.941890 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:43.925742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:43.941890 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:43.930313    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-218.ec2.internal\" not found" node="ip-10-0-132-218.ec2.internal"
Jul 16 17:33:44.005835 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.005805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9544c8d8a199e4cc63eaab616f80015e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal\" (UID: \"9544c8d8a199e4cc63eaab616f80015e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:44.005956 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.005847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9544c8d8a199e4cc63eaab616f80015e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal\" (UID: \"9544c8d8a199e4cc63eaab616f80015e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:44.005956 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.005873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9544c8d8a199e4cc63eaab616f80015e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal\" (UID: \"9544c8d8a199e4cc63eaab616f80015e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:44.005956 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.005888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9544c8d8a199e4cc63eaab616f80015e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal\" (UID: \"9544c8d8a199e4cc63eaab616f80015e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:44.020919 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:44.020888    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:44.121604 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:44.121575    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:44.222014 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:44.221972    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:44.233111 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.233091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:44.322698 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:44.322618    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:44.387903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.387881    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 17:33:44.388018 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.388001    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 17:33:44.423193 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:44.423170    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:44.474737 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.474699    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 17:28:43 +0000 UTC" deadline="2028-01-10 21:13:29.626052493 +0000 UTC"
Jul 16 17:33:44.474737 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.474729    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13035h39m45.151326637s"
Jul 16 17:33:44.523566 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:44.523530    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:44.540543 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.540518    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:33:44.617400 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.617337    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:33:44.624043 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:44.624024    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:44.724271 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:44.724235    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-218.ec2.internal\" not found"
Jul 16 17:33:44.740748 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:44.740716    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9544c8d8a199e4cc63eaab616f80015e.slice/crio-9e4086c313dc071bef616aad70e33cdd6aa79b5aba0754f0ce26f75f26d7f375 WatchSource:0}: Error finding container 9e4086c313dc071bef616aad70e33cdd6aa79b5aba0754f0ce26f75f26d7f375: Status 404 returned error can't find the container with id 9e4086c313dc071bef616aad70e33cdd6aa79b5aba0754f0ce26f75f26d7f375
Jul 16 17:33:44.740995 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:44.740979    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7aea08f316a1dd14e58ba5cd1a755a.slice/crio-99b6860506236572074f297ec7cfb0054b51f1d34eb3e2a3f003a56f3788d561 WatchSource:0}: Error finding container 99b6860506236572074f297ec7cfb0054b51f1d34eb3e2a3f003a56f3788d561: Status 404 returned error can't find the container with id 99b6860506236572074f297ec7cfb0054b51f1d34eb3e2a3f003a56f3788d561
Jul 16 17:33:44.745638 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.745619    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:33:44.753645 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.753626    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:33:44.769699 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.769674    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:44.788857 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.788835    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 17:33:44.789679 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.789667    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal"
Jul 16 17:33:44.802656 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:44.802639    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 17:33:45.441840 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.441818    2563 apiserver.go:52] "Watching apiserver"
Jul 16 17:33:45.453281 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.451597    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 17:33:45.454519 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.454488    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rsfgv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt","openshift-cluster-node-tuning-operator/tuned-949cg","openshift-multus/multus-whxjd","openshift-multus/network-metrics-daemon-gn4pn","openshift-network-operator/iptables-alerter-wnvg9","openshift-ovn-kubernetes/ovnkube-node-2l4ts","kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal","openshift-dns/node-resolver-mxzqv","openshift-image-registry/node-ca-vdpc9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal","openshift-multus/multus-additional-cni-plugins-25sdp","openshift-network-diagnostics/network-check-target-hr44p","kube-system/global-pull-secret-syncer-tnzn7"]
Jul 16 17:33:45.457360 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.457164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.459436 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.459413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:45.459567 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.459506    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:33:45.461654 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.461635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.463774 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.463750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.465979 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.465961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:45.466059 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.466034    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:33:45.468169 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.468152    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.470453 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.470249    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 17:33:45.470453 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.470377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.472600 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.472581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:33:45.474768 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.474448    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 17:33:45.474768 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.474479    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 17:33:45.474768 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.474503    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 17:33:45.474768 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.474479    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 17:33:45.475405 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.474883    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 17:33:45.475405 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.475034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.475747 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.475729    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8s6w\""
Jul 16 17:33:45.476125 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.476103    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 17:33:45.476197 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.476146    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:33:45.476985 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.476967    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 17:33:45.477646 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.477300    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 17:33:45.477646 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.477330    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 17:33:45.477646 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.477399    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 17:33:45.477646 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.477499    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bx5hb\""
Jul 16 17:33:45.477646 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.477521    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 17:33:45.477646 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.477620    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bjpk6\""
Jul 16 17:33:45.477960 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.477785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.477960 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.477886    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 17:33:45.478345 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.478326    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7fs69\""
Jul 16 17:33:45.478539 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.478523    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:33:45.479158 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.479135    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fqqxp\""
Jul 16 17:33:45.480138 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.480107    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:45.480258 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.480224    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:33:45.480738 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.480722    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 17:33:45.482981 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.482958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.485348 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.485329    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 17:33:45.485896 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.485877    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 17:33:45.489424 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.489408    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 17:33:45.494455 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.494434    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 17:33:45.494814 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.494794    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 17:33:45.495036 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.495021    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 17:33:45.499420 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.499398    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 17:33:45.499950 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.499927    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-scdbc\""
Jul 16 17:33:45.500329 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.500311    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 17:33:45.500490 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.500471    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-54fnn\""
Jul 16 17:33:45.500572 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.500544    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 17:33:45.500736 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.500719    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 17:33:45.500862 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.500846    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6cv4n\""
Jul 16 17:33:45.503151 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.503133    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hjxg2\""
Jul 16 17:33:45.503434 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.503417    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 17:33:45.503980 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.503963    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 17:33:45.513988 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.513964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63d63b93-8c90-401b-ba2c-1585ea70fb35-tmp\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.514076 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514076 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-cnibin\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514076 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-daemon-config\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514188 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-var-lib-kubelet\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.514188 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-conf-dir\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514188 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26qrw\" (UniqueName: \"kubernetes.io/projected/fa1f394b-ac23-4640-86ed-75f04466fa1e-kube-api-access-26qrw\") pod \"node-resolver-mxzqv\" (UID: \"fa1f394b-ac23-4640-86ed-75f04466fa1e\") " pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.514307 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-etc-openvswitch\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514307 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-node-log\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514307 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffe1aeb1-9246-430c-964b-2eb6db9dd017-env-overrides\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514307 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-hostroot\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514307 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lhp\" (UniqueName: \"kubernetes.io/projected/4a921cd4-90af-41b4-828f-9c74ce755d39-kube-api-access-89lhp\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-tuned\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.514476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-var-lib-cni-multus\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-lib-modules\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.514476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-run-ovn\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-socket-dir-parent\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-sysctl-conf\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.514739 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-systemd-units\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514739 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-var-lib-openvswitch\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514739 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-cni-bin\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514739 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffe1aeb1-9246-430c-964b-2eb6db9dd017-ovnkube-config\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514739 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-var-lib-kubelet\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514739 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-sysconfig\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.514739 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-etc-kubernetes\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514739 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-modprobe-d\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-sys\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-host\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-var-lib-cni-bin\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-run-multus-certs\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkzk9\" (UniqueName: \"kubernetes.io/projected/2f7cb219-cff0-464c-b353-97537f8d29c3-kube-api-access-hkzk9\") pod \"iptables-alerter-wnvg9\" (UID: \"2f7cb219-cff0-464c-b353-97537f8d29c3\") " pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa1f394b-ac23-4640-86ed-75f04466fa1e-hosts-file\") pod \"node-resolver-mxzqv\" (UID: \"fa1f394b-ac23-4640-86ed-75f04466fa1e\") " pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-run-netns\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffe1aeb1-9246-430c-964b-2eb6db9dd017-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffe1aeb1-9246-430c-964b-2eb6db9dd017-ovnkube-script-lib\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.514994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.514982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a921cd4-90af-41b4-828f-9c74ce755d39-cni-binary-copy\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-systemd\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-run-openvswitch\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-cni-netd\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7m98\" (UniqueName: \"kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98\") pod \"network-check-target-hr44p\" (UID: \"33b91431-5098-4f19-9e2f-489917539c2f\") " pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-kubernetes\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-run\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm52v\" (UniqueName: \"kubernetes.io/projected/ffe1aeb1-9246-430c-964b-2eb6db9dd017-kube-api-access-rm52v\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhph\" (UniqueName: \"kubernetes.io/projected/c928af6c-6ba7-40bb-8e7a-4a418754b963-kube-api-access-jdhph\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-run-systemd\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f7cb219-cff0-464c-b353-97537f8d29c3-host-slash\") pod \"iptables-alerter-wnvg9\" (UID: \"2f7cb219-cff0-464c-b353-97537f8d29c3\") " pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-sysctl-d\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgmqm\" (UniqueName: \"kubernetes.io/projected/63d63b93-8c90-401b-ba2c-1585ea70fb35-kube-api-access-hgmqm\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.515332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-system-cni-dir\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.515879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f7cb219-cff0-464c-b353-97537f8d29c3-iptables-alerter-script\") pod \"iptables-alerter-wnvg9\" (UID: \"2f7cb219-cff0-464c-b353-97537f8d29c3\") " pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.515879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1f394b-ac23-4640-86ed-75f04466fa1e-tmp-dir\") pod \"node-resolver-mxzqv\" (UID: \"fa1f394b-ac23-4640-86ed-75f04466fa1e\") " pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.515879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-cni-dir\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.515879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-os-release\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.515879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-run-k8s-cni-cncf-io\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.515879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:45.515879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-slash\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.515879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-log-socket\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.515879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-run-netns\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.515879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.515660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-kubelet\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.527883 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.527863    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 17:33:45.556707 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.556676    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vtdnp"
Jul 16 17:33:45.569184 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.569154    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vtdnp"
Jul 16 17:33:45.570970 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.570945    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 17:33:45.616785 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.616752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-cni-netd\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.616994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.616800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7m98\" (UniqueName: \"kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98\") pod \"network-check-target-hr44p\" (UID: \"33b91431-5098-4f19-9e2f-489917539c2f\") " pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:45.616994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.616829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04ee78ee-1c82-4394-8b56-23cda890ae80-host\") pod \"node-ca-vdpc9\" (UID: \"04ee78ee-1c82-4394-8b56-23cda890ae80\") " pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.616994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.616862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-os-release\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.616994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.616888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-kubernetes\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.616994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.616909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-run\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.616994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.616932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm52v\" (UniqueName: \"kubernetes.io/projected/ffe1aeb1-9246-430c-964b-2eb6db9dd017-kube-api-access-rm52v\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.616994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.616955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhph\" (UniqueName: \"kubernetes.io/projected/c928af6c-6ba7-40bb-8e7a-4a418754b963-kube-api-access-jdhph\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:45.616994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.616979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef3ddfba-d14b-493c-8c03-0c96efeb611b-kubelet-config\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-run-systemd\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f7cb219-cff0-464c-b353-97537f8d29c3-host-slash\") pod \"iptables-alerter-wnvg9\" (UID: \"2f7cb219-cff0-464c-b353-97537f8d29c3\") " pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-sysctl-d\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgmqm\" (UniqueName: \"kubernetes.io/projected/63d63b93-8c90-401b-ba2c-1585ea70fb35-kube-api-access-hgmqm\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-system-cni-dir\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482ct\" (UniqueName: \"kubernetes.io/projected/dea026da-38bb-4bf7-9382-7405d0abc6d4-kube-api-access-482ct\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f7cb219-cff0-464c-b353-97537f8d29c3-iptables-alerter-script\") pod \"iptables-alerter-wnvg9\" (UID: \"2f7cb219-cff0-464c-b353-97537f8d29c3\") " pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1f394b-ac23-4640-86ed-75f04466fa1e-tmp-dir\") pod \"node-resolver-mxzqv\" (UID: \"fa1f394b-ac23-4640-86ed-75f04466fa1e\") " pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-cni-dir\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-os-release\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-run-k8s-cni-cncf-io\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef3ddfba-d14b-493c-8c03-0c96efeb611b-dbus\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-slash\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-registration-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.617395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-device-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-cnibin\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-log-socket\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-run-netns\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-system-cni-dir\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-kubelet\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-sys-fs\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63d63b93-8c90-401b-ba2c-1585ea70fb35-tmp\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-run\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-cnibin\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-daemon-config\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-cnibin\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-cni-netd\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-kubelet\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.617991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-kubernetes\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-slash\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-log-socket\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.618248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-run-netns\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618224    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-run-systemd\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d656ee83-7adb-4ed3-8255-215c6736b17a-agent-certs\") pod \"konnectivity-agent-rsfgv\" (UID: \"d656ee83-7adb-4ed3-8255-215c6736b17a\") " pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-system-cni-dir\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1f394b-ac23-4640-86ed-75f04466fa1e-tmp-dir\") pod \"node-resolver-mxzqv\" (UID: \"fa1f394b-ac23-4640-86ed-75f04466fa1e\") " pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-run-k8s-cni-cncf-io\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-cni-dir\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f7cb219-cff0-464c-b353-97537f8d29c3-host-slash\") pod \"iptables-alerter-wnvg9\" (UID: \"2f7cb219-cff0-464c-b353-97537f8d29c3\") " pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-var-lib-kubelet\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-os-release\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-sysctl-d\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-conf-dir\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-conf-dir\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-socket-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-var-lib-kubelet\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.619074 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dea026da-38bb-4bf7-9382-7405d0abc6d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-daemon-config\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f7cb219-cff0-464c-b353-97537f8d29c3-iptables-alerter-script\") pod \"iptables-alerter-wnvg9\" (UID: \"2f7cb219-cff0-464c-b353-97537f8d29c3\") " pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26qrw\" (UniqueName: \"kubernetes.io/projected/fa1f394b-ac23-4640-86ed-75f04466fa1e-kube-api-access-26qrw\") pod \"node-resolver-mxzqv\" (UID: \"fa1f394b-ac23-4640-86ed-75f04466fa1e\") " pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-etc-openvswitch\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.618905    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-node-log\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffe1aeb1-9246-430c-964b-2eb6db9dd017-env-overrides\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-etc-openvswitch\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.618996    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs podName:c928af6c-6ba7-40bb-8e7a-4a418754b963 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:46.118956457 +0000 UTC m=+3.105492054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs") pod "network-metrics-daemon-gn4pn" (UID: "c928af6c-6ba7-40bb-8e7a-4a418754b963") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-hostroot\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lhp\" (UniqueName: \"kubernetes.io/projected/4a921cd4-90af-41b4-828f-9c74ce755d39-kube-api-access-89lhp\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqln4\" (UniqueName: \"kubernetes.io/projected/07d6c3ec-f524-4f27-98a6-2e0622dd3926-kube-api-access-pqln4\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-hostroot\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-tuned\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.618994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-node-log\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-var-lib-cni-multus\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.619935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dea026da-38bb-4bf7-9382-7405d0abc6d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-lib-modules\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-run-ovn\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-socket-dir-parent\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-sysctl-conf\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-multus-socket-dir-parent\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffe1aeb1-9246-430c-964b-2eb6db9dd017-env-overrides\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-var-lib-cni-multus\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-run-ovn\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-systemd-units\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-sysctl-conf\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-lib-modules\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-var-lib-openvswitch\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-systemd-units\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-cni-bin\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-var-lib-openvswitch\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.620776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffe1aeb1-9246-430c-964b-2eb6db9dd017-ovnkube-config\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-cni-bin\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-var-lib-kubelet\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04ee78ee-1c82-4394-8b56-23cda890ae80-serviceca\") pod \"node-ca-vdpc9\" (UID: \"04ee78ee-1c82-4394-8b56-23cda890ae80\") " pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-var-lib-kubelet\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjxdg\" (UniqueName: \"kubernetes.io/projected/04ee78ee-1c82-4394-8b56-23cda890ae80-kube-api-access-wjxdg\") pod \"node-ca-vdpc9\" (UID: \"04ee78ee-1c82-4394-8b56-23cda890ae80\") " pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-sysconfig\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-etc-kubernetes\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-sysconfig\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d656ee83-7adb-4ed3-8255-215c6736b17a-konnectivity-ca\") pod \"konnectivity-agent-rsfgv\" (UID: \"d656ee83-7adb-4ed3-8255-215c6736b17a\") " pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-etc-kubernetes\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.619990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dea026da-38bb-4bf7-9382-7405d0abc6d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-modprobe-d\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-sys\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-host\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-var-lib-cni-bin\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-modprobe-d\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.621457 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-run-multus-certs\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkzk9\" (UniqueName: \"kubernetes.io/projected/2f7cb219-cff0-464c-b353-97537f8d29c3-kube-api-access-hkzk9\") pod \"iptables-alerter-wnvg9\" (UID: \"2f7cb219-cff0-464c-b353-97537f8d29c3\") " pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-host\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffe1aeb1-9246-430c-964b-2eb6db9dd017-ovnkube-config\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa1f394b-ac23-4640-86ed-75f04466fa1e-hosts-file\") pod \"node-resolver-mxzqv\" (UID: \"fa1f394b-ac23-4640-86ed-75f04466fa1e\") " pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-run-netns\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-sys\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa1f394b-ac23-4640-86ed-75f04466fa1e-hosts-file\") pod \"node-resolver-mxzqv\" (UID: \"fa1f394b-ac23-4640-86ed-75f04466fa1e\") " pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffe1aeb1-9246-430c-964b-2eb6db9dd017-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffe1aeb1-9246-430c-964b-2eb6db9dd017-ovnkube-script-lib\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-run-netns\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a921cd4-90af-41b4-828f-9c74ce755d39-cni-binary-copy\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-etc-selinux\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-run-multus-certs\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a921cd4-90af-41b4-828f-9c74ce755d39-host-var-lib-cni-bin\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-systemd\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.622302 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-run-openvswitch\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.622900 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.622900 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-systemd\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.622900 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.620782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe1aeb1-9246-430c-964b-2eb6db9dd017-run-openvswitch\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.622900 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.621363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a921cd4-90af-41b4-828f-9c74ce755d39-cni-binary-copy\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.622900 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.621719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffe1aeb1-9246-430c-964b-2eb6db9dd017-ovnkube-script-lib\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.623139 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.623051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63d63b93-8c90-401b-ba2c-1585ea70fb35-tmp\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.623247 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.623167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffe1aeb1-9246-430c-964b-2eb6db9dd017-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.624595 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.624571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63d63b93-8c90-401b-ba2c-1585ea70fb35-etc-tuned\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.648999 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.648905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lhp\" (UniqueName: \"kubernetes.io/projected/4a921cd4-90af-41b4-828f-9c74ce755d39-kube-api-access-89lhp\") pod \"multus-whxjd\" (UID: \"4a921cd4-90af-41b4-828f-9c74ce755d39\") " pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.654850 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.654824    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:33:45.654992 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.654857    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:33:45.654992 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.654871    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7m98 for pod openshift-network-diagnostics/network-check-target-hr44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:33:45.654992 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.654941    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98 podName:33b91431-5098-4f19-9e2f-489917539c2f nodeName:}" failed. No retries permitted until 2026-07-16 17:33:46.154919474 +0000 UTC m=+3.141455075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d7m98" (UniqueName: "kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98") pod "network-check-target-hr44p" (UID: "33b91431-5098-4f19-9e2f-489917539c2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:33:45.659956 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.659924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkzk9\" (UniqueName: \"kubernetes.io/projected/2f7cb219-cff0-464c-b353-97537f8d29c3-kube-api-access-hkzk9\") pod \"iptables-alerter-wnvg9\" (UID: \"2f7cb219-cff0-464c-b353-97537f8d29c3\") " pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.661128 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.660798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26qrw\" (UniqueName: \"kubernetes.io/projected/fa1f394b-ac23-4640-86ed-75f04466fa1e-kube-api-access-26qrw\") pod \"node-resolver-mxzqv\" (UID: \"fa1f394b-ac23-4640-86ed-75f04466fa1e\") " pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.661128 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.661074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgmqm\" (UniqueName: \"kubernetes.io/projected/63d63b93-8c90-401b-ba2c-1585ea70fb35-kube-api-access-hgmqm\") pod \"tuned-949cg\" (UID: \"63d63b93-8c90-401b-ba2c-1585ea70fb35\") " pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.663184 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.663162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm52v\" (UniqueName: \"kubernetes.io/projected/ffe1aeb1-9246-430c-964b-2eb6db9dd017-kube-api-access-rm52v\") pod \"ovnkube-node-2l4ts\" (UID: \"ffe1aeb1-9246-430c-964b-2eb6db9dd017\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.664933 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.664890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhph\" (UniqueName: \"kubernetes.io/projected/c928af6c-6ba7-40bb-8e7a-4a418754b963-kube-api-access-jdhph\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:45.674454 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.674372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal" event={"ID":"9544c8d8a199e4cc63eaab616f80015e","Type":"ContainerStarted","Data":"9e4086c313dc071bef616aad70e33cdd6aa79b5aba0754f0ce26f75f26d7f375"}
Jul 16 17:33:45.676029 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.676002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal" event={"ID":"2b7aea08f316a1dd14e58ba5cd1a755a","Type":"ContainerStarted","Data":"99b6860506236572074f297ec7cfb0054b51f1d34eb3e2a3f003a56f3788d561"}
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d656ee83-7adb-4ed3-8255-215c6736b17a-agent-certs\") pod \"konnectivity-agent-rsfgv\" (UID: \"d656ee83-7adb-4ed3-8255-215c6736b17a\") " pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-socket-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dea026da-38bb-4bf7-9382-7405d0abc6d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqln4\" (UniqueName: \"kubernetes.io/projected/07d6c3ec-f524-4f27-98a6-2e0622dd3926-kube-api-access-pqln4\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dea026da-38bb-4bf7-9382-7405d0abc6d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04ee78ee-1c82-4394-8b56-23cda890ae80-serviceca\") pod \"node-ca-vdpc9\" (UID: \"04ee78ee-1c82-4394-8b56-23cda890ae80\") " pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjxdg\" (UniqueName: \"kubernetes.io/projected/04ee78ee-1c82-4394-8b56-23cda890ae80-kube-api-access-wjxdg\") pod \"node-ca-vdpc9\" (UID: \"04ee78ee-1c82-4394-8b56-23cda890ae80\") " pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d656ee83-7adb-4ed3-8255-215c6736b17a-konnectivity-ca\") pod \"konnectivity-agent-rsfgv\" (UID: \"d656ee83-7adb-4ed3-8255-215c6736b17a\") " pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dea026da-38bb-4bf7-9382-7405d0abc6d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-etc-selinux\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04ee78ee-1c82-4394-8b56-23cda890ae80-host\") pod \"node-ca-vdpc9\" (UID: \"04ee78ee-1c82-4394-8b56-23cda890ae80\") " pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-os-release\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef3ddfba-d14b-493c-8c03-0c96efeb611b-kubelet-config\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482ct\" (UniqueName: \"kubernetes.io/projected/dea026da-38bb-4bf7-9382-7405d0abc6d4-kube-api-access-482ct\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.722545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef3ddfba-d14b-493c-8c03-0c96efeb611b-dbus\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-registration-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-device-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-cnibin\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-system-cni-dir\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-sys-fs\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.721751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-sys-fs\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.722271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-etc-selinux\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.722425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.722511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-socket-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.723041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dea026da-38bb-4bf7-9382-7405d0abc6d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.723526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.723340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.724178 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.723805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dea026da-38bb-4bf7-9382-7405d0abc6d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.724237 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.724205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04ee78ee-1c82-4394-8b56-23cda890ae80-serviceca\") pod \"node-ca-vdpc9\" (UID: \"04ee78ee-1c82-4394-8b56-23cda890ae80\") " pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.724898 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.724865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dea026da-38bb-4bf7-9382-7405d0abc6d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.725009 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.724901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d656ee83-7adb-4ed3-8255-215c6736b17a-konnectivity-ca\") pod \"konnectivity-agent-rsfgv\" (UID: \"d656ee83-7adb-4ed3-8255-215c6736b17a\") " pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:33:45.725009 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.724954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef3ddfba-d14b-493c-8c03-0c96efeb611b-dbus\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:45.725009 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.724958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-device-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.725009 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.724995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04ee78ee-1c82-4394-8b56-23cda890ae80-host\") pod \"node-ca-vdpc9\" (UID: \"04ee78ee-1c82-4394-8b56-23cda890ae80\") " pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.725166 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.725031    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:33:45.725166 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.725066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-os-release\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.725166 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:45.725089    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret podName:ef3ddfba-d14b-493c-8c03-0c96efeb611b nodeName:}" failed. No retries permitted until 2026-07-16 17:33:46.225070349 +0000 UTC m=+3.211605936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret") pod "global-pull-secret-syncer-tnzn7" (UID: "ef3ddfba-d14b-493c-8c03-0c96efeb611b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:33:45.725166 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.725119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef3ddfba-d14b-493c-8c03-0c96efeb611b-kubelet-config\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:45.725166 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.725131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07d6c3ec-f524-4f27-98a6-2e0622dd3926-registration-dir\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.725385 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.725318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-cnibin\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.725385 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.725358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dea026da-38bb-4bf7-9382-7405d0abc6d4-system-cni-dir\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.725761 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.725738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d656ee83-7adb-4ed3-8255-215c6736b17a-agent-certs\") pod \"konnectivity-agent-rsfgv\" (UID: \"d656ee83-7adb-4ed3-8255-215c6736b17a\") " pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:33:45.747411 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.747348    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjxdg\" (UniqueName: \"kubernetes.io/projected/04ee78ee-1c82-4394-8b56-23cda890ae80-kube-api-access-wjxdg\") pod \"node-ca-vdpc9\" (UID: \"04ee78ee-1c82-4394-8b56-23cda890ae80\") " pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.748638 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.748609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqln4\" (UniqueName: \"kubernetes.io/projected/07d6c3ec-f524-4f27-98a6-2e0622dd3926-kube-api-access-pqln4\") pod \"aws-ebs-csi-driver-node-dglpt\" (UID: \"07d6c3ec-f524-4f27-98a6-2e0622dd3926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.748774 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.748699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-482ct\" (UniqueName: \"kubernetes.io/projected/dea026da-38bb-4bf7-9382-7405d0abc6d4-kube-api-access-482ct\") pod \"multus-additional-cni-plugins-25sdp\" (UID: \"dea026da-38bb-4bf7-9382-7405d0abc6d4\") " pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.768385 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.768180    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:33:45.775538 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.775243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-949cg"
Jul 16 17:33:45.786113 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.785915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-whxjd"
Jul 16 17:33:45.792161 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.792120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wnvg9"
Jul 16 17:33:45.797926 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.797897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mxzqv"
Jul 16 17:33:45.806806 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.806770    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:33:45.815649 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.815626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt"
Jul 16 17:33:45.823325 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.823158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vdpc9"
Jul 16 17:33:45.829866 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.829845    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-25sdp"
Jul 16 17:33:45.914848 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:45.914805    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:33:46.124630 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.124529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:46.124784 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:46.124697    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:33:46.124784 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:46.124771    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs podName:c928af6c-6ba7-40bb-8e7a-4a418754b963 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:47.124751236 +0000 UTC m=+4.111286819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs") pod "network-metrics-daemon-gn4pn" (UID: "c928af6c-6ba7-40bb-8e7a-4a418754b963") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:33:46.225194 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.225157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7m98\" (UniqueName: \"kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98\") pod \"network-check-target-hr44p\" (UID: \"33b91431-5098-4f19-9e2f-489917539c2f\") " pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:46.225358 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.225217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:46.225358 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:46.225320    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:33:46.225358 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:46.225342    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:33:46.225515 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:46.225372    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:33:46.225515 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:46.225381    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret podName:ef3ddfba-d14b-493c-8c03-0c96efeb611b nodeName:}" failed. No retries permitted until 2026-07-16 17:33:47.225363957 +0000 UTC m=+4.211899542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret") pod "global-pull-secret-syncer-tnzn7" (UID: "ef3ddfba-d14b-493c-8c03-0c96efeb611b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:33:46.225515 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:46.225386    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7m98 for pod openshift-network-diagnostics/network-check-target-hr44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:33:46.225515 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:46.225439    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98 podName:33b91431-5098-4f19-9e2f-489917539c2f nodeName:}" failed. No retries permitted until 2026-07-16 17:33:47.225423054 +0000 UTC m=+4.211958654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7m98" (UniqueName: "kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98") pod "network-check-target-hr44p" (UID: "33b91431-5098-4f19-9e2f-489917539c2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:33:46.573389 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.573352    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:28:45 +0000 UTC" deadline="2028-02-17 15:18:36.15018464 +0000 UTC"
Jul 16 17:33:46.573389 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.573379    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13941h44m49.576808966s"
Jul 16 17:33:46.607639 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:46.607611    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd656ee83_7adb_4ed3_8255_215c6736b17a.slice/crio-6c3d42c5e5481cf1396112e5539325ab6f4350eebcbdb61074a2466b47151936 WatchSource:0}: Error finding container 6c3d42c5e5481cf1396112e5539325ab6f4350eebcbdb61074a2466b47151936: Status 404 returned error can't find the container with id 6c3d42c5e5481cf1396112e5539325ab6f4350eebcbdb61074a2466b47151936
Jul 16 17:33:46.609132 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:46.609102    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a921cd4_90af_41b4_828f_9c74ce755d39.slice/crio-58fb263177245c13c867d315fc6b93d6a202000ea4a0c655c7c7ba5e24853b76 WatchSource:0}: Error finding container 58fb263177245c13c867d315fc6b93d6a202000ea4a0c655c7c7ba5e24853b76: Status 404 returned error can't find the container with id 58fb263177245c13c867d315fc6b93d6a202000ea4a0c655c7c7ba5e24853b76
Jul 16 17:33:46.610138 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:46.609916    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ee78ee_1c82_4394_8b56_23cda890ae80.slice/crio-724eca57dccd7f36a538457f3130550cb78b6cdb58d25519364bdeb82c34115c WatchSource:0}: Error finding container 724eca57dccd7f36a538457f3130550cb78b6cdb58d25519364bdeb82c34115c: Status 404 returned error can't find the container with id 724eca57dccd7f36a538457f3130550cb78b6cdb58d25519364bdeb82c34115c
Jul 16 17:33:46.613969 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:46.613946    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d63b93_8c90_401b_ba2c_1585ea70fb35.slice/crio-1fc280acd792d94f789f5edf9fedf889a8ef759dffe07408d8df5669f5e2a676 WatchSource:0}: Error finding container 1fc280acd792d94f789f5edf9fedf889a8ef759dffe07408d8df5669f5e2a676: Status 404 returned error can't find the container with id 1fc280acd792d94f789f5edf9fedf889a8ef759dffe07408d8df5669f5e2a676
Jul 16 17:33:46.614600 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:46.614542    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1f394b_ac23_4640_86ed_75f04466fa1e.slice/crio-bc915acbd8285b5516b961740c56becf0428a6e20a522f8c599630972d1656e4 WatchSource:0}: Error finding container bc915acbd8285b5516b961740c56becf0428a6e20a522f8c599630972d1656e4: Status 404 returned error can't find the container with id bc915acbd8285b5516b961740c56becf0428a6e20a522f8c599630972d1656e4
Jul 16 17:33:46.637465 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:46.637440    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe1aeb1_9246_430c_964b_2eb6db9dd017.slice/crio-dba7723e266c9d7ecf6528cce7a2d92d84f4c87ecf2f41e3b1f0a687a2bfbe14 WatchSource:0}: Error finding container dba7723e266c9d7ecf6528cce7a2d92d84f4c87ecf2f41e3b1f0a687a2bfbe14: Status 404 returned error can't find the container with id dba7723e266c9d7ecf6528cce7a2d92d84f4c87ecf2f41e3b1f0a687a2bfbe14
Jul 16 17:33:46.638689 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:46.638665    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea026da_38bb_4bf7_9382_7405d0abc6d4.slice/crio-7429c4f022353b112a5ccabcdecf0b0291f65b668ecdbfeee586fc9d10696254 WatchSource:0}: Error finding container 7429c4f022353b112a5ccabcdecf0b0291f65b668ecdbfeee586fc9d10696254: Status 404 returned error can't find the container with id 7429c4f022353b112a5ccabcdecf0b0291f65b668ecdbfeee586fc9d10696254
Jul 16 17:33:46.639337 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:46.639301    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d6c3ec_f524_4f27_98a6_2e0622dd3926.slice/crio-eaa64cb7122d55fcca1062f9534b03c25c2e4e835ffd55b3f77fa3b3c6d0c763 WatchSource:0}: Error finding container eaa64cb7122d55fcca1062f9534b03c25c2e4e835ffd55b3f77fa3b3c6d0c763: Status 404 returned error can't find the container with id eaa64cb7122d55fcca1062f9534b03c25c2e4e835ffd55b3f77fa3b3c6d0c763
Jul 16 17:33:46.640156 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:33:46.640123    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7cb219_cff0_464c_b353_97537f8d29c3.slice/crio-7562458940f254ab431a3614c2d08ec00234439f1d11bccecfcaaaa9e6aee9d0 WatchSource:0}: Error finding container 7562458940f254ab431a3614c2d08ec00234439f1d11bccecfcaaaa9e6aee9d0: Status 404 returned error can't find the container with id 7562458940f254ab431a3614c2d08ec00234439f1d11bccecfcaaaa9e6aee9d0
Jul 16 17:33:46.681541 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.681510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-whxjd" event={"ID":"4a921cd4-90af-41b4-828f-9c74ce755d39","Type":"ContainerStarted","Data":"58fb263177245c13c867d315fc6b93d6a202000ea4a0c655c7c7ba5e24853b76"}
Jul 16 17:33:46.682529 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.682508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vdpc9" event={"ID":"04ee78ee-1c82-4394-8b56-23cda890ae80","Type":"ContainerStarted","Data":"724eca57dccd7f36a538457f3130550cb78b6cdb58d25519364bdeb82c34115c"}
Jul 16 17:33:46.683395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.683376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rsfgv" event={"ID":"d656ee83-7adb-4ed3-8255-215c6736b17a","Type":"ContainerStarted","Data":"6c3d42c5e5481cf1396112e5539325ab6f4350eebcbdb61074a2466b47151936"}
Jul 16 17:33:46.684499 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.684474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wnvg9" event={"ID":"2f7cb219-cff0-464c-b353-97537f8d29c3","Type":"ContainerStarted","Data":"7562458940f254ab431a3614c2d08ec00234439f1d11bccecfcaaaa9e6aee9d0"}
Jul 16 17:33:46.685563 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.685530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt" event={"ID":"07d6c3ec-f524-4f27-98a6-2e0622dd3926","Type":"ContainerStarted","Data":"eaa64cb7122d55fcca1062f9534b03c25c2e4e835ffd55b3f77fa3b3c6d0c763"}
Jul 16 17:33:46.686532 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.686515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" event={"ID":"ffe1aeb1-9246-430c-964b-2eb6db9dd017","Type":"ContainerStarted","Data":"dba7723e266c9d7ecf6528cce7a2d92d84f4c87ecf2f41e3b1f0a687a2bfbe14"}
Jul 16 17:33:46.687445 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.687428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mxzqv" event={"ID":"fa1f394b-ac23-4640-86ed-75f04466fa1e","Type":"ContainerStarted","Data":"bc915acbd8285b5516b961740c56becf0428a6e20a522f8c599630972d1656e4"}
Jul 16 17:33:46.688475 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.688442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sdp" event={"ID":"dea026da-38bb-4bf7-9382-7405d0abc6d4","Type":"ContainerStarted","Data":"7429c4f022353b112a5ccabcdecf0b0291f65b668ecdbfeee586fc9d10696254"}
Jul 16 17:33:46.689623 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:46.689601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-949cg" event={"ID":"63d63b93-8c90-401b-ba2c-1585ea70fb35","Type":"ContainerStarted","Data":"1fc280acd792d94f789f5edf9fedf889a8ef759dffe07408d8df5669f5e2a676"}
Jul 16 17:33:47.132664 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.132339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:47.132664 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.132618    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:33:47.132879 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.132702    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs podName:c928af6c-6ba7-40bb-8e7a-4a418754b963 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:49.132682882 +0000 UTC m=+6.119218480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs") pod "network-metrics-daemon-gn4pn" (UID: "c928af6c-6ba7-40bb-8e7a-4a418754b963") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:33:47.233993 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.233959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:47.234218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.234042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7m98\" (UniqueName: \"kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98\") pod \"network-check-target-hr44p\" (UID: \"33b91431-5098-4f19-9e2f-489917539c2f\") " pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:47.234218 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.234206    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:33:47.234333 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.234234    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:33:47.234333 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.234248    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7m98 for pod openshift-network-diagnostics/network-check-target-hr44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:33:47.234333 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.234310    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98 podName:33b91431-5098-4f19-9e2f-489917539c2f nodeName:}" failed. No retries permitted until 2026-07-16 17:33:49.234292465 +0000 UTC m=+6.220828055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7m98" (UniqueName: "kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98") pod "network-check-target-hr44p" (UID: "33b91431-5098-4f19-9e2f-489917539c2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:33:47.234736 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.234717    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:33:47.234803 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.234783    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret podName:ef3ddfba-d14b-493c-8c03-0c96efeb611b nodeName:}" failed. No retries permitted until 2026-07-16 17:33:49.234768622 +0000 UTC m=+6.221304208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret") pod "global-pull-secret-syncer-tnzn7" (UID: "ef3ddfba-d14b-493c-8c03-0c96efeb611b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:33:47.574254 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.574207    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:28:45 +0000 UTC" deadline="2028-01-02 19:54:25.795551888 +0000 UTC"
Jul 16 17:33:47.574254 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.574253    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12842h20m38.221303885s"
Jul 16 17:33:47.670736 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.670694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:47.670897 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.670844    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:33:47.671263 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.671239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:47.671381 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.671346    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:33:47.671452 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.671439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:47.671531 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:47.671515    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:33:47.704899 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.704815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal" event={"ID":"2b7aea08f316a1dd14e58ba5cd1a755a","Type":"ContainerStarted","Data":"b9ab2f5fe3d910cfdfc191711c05d2f86fa922b38c00055d31ebbf55f54bb9b7"}
Jul 16 17:33:47.717297 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.716675    2563 generic.go:356] "Generic (PLEG): container finished" podID="9544c8d8a199e4cc63eaab616f80015e" containerID="8677bcbae9d35c11e39d0f1977336d1e2fad0033ead65d8b18bb2eaba35c8a75" exitCode=0
Jul 16 17:33:47.717297 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.716729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal" event={"ID":"9544c8d8a199e4cc63eaab616f80015e","Type":"ContainerDied","Data":"8677bcbae9d35c11e39d0f1977336d1e2fad0033ead65d8b18bb2eaba35c8a75"}
Jul 16 17:33:47.726837 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:47.726542    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-218.ec2.internal" podStartSLOduration=3.7265218879999997 podStartE2EDuration="3.726521888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:47.726431324 +0000 UTC m=+4.712966946" watchObservedRunningTime="2026-07-16 17:33:47.726521888 +0000 UTC m=+4.713057497"
Jul 16 17:33:48.732176 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:48.732106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal" event={"ID":"9544c8d8a199e4cc63eaab616f80015e","Type":"ContainerStarted","Data":"d66254de106c14312708f739302365a2774d6206449660403a63e1c1041fd973"}
Jul 16 17:33:49.149472 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:49.148768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:49.149472 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.148943    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:33:49.149472 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.149006    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs podName:c928af6c-6ba7-40bb-8e7a-4a418754b963 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:53.148987694 +0000 UTC m=+10.135523294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs") pod "network-metrics-daemon-gn4pn" (UID: "c928af6c-6ba7-40bb-8e7a-4a418754b963") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:33:49.249414 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:49.249377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7m98\" (UniqueName: \"kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98\") pod \"network-check-target-hr44p\" (UID: \"33b91431-5098-4f19-9e2f-489917539c2f\") " pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:49.249614 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:49.249451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:49.249691 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.249666    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:33:49.249751 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.249740    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret podName:ef3ddfba-d14b-493c-8c03-0c96efeb611b nodeName:}" failed. No retries permitted until 2026-07-16 17:33:53.249720981 +0000 UTC m=+10.236256579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret") pod "global-pull-secret-syncer-tnzn7" (UID: "ef3ddfba-d14b-493c-8c03-0c96efeb611b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:33:49.250160 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.250141    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:33:49.250237 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.250166    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:33:49.250237 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.250180    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7m98 for pod openshift-network-diagnostics/network-check-target-hr44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:33:49.250237 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.250227    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98 podName:33b91431-5098-4f19-9e2f-489917539c2f nodeName:}" failed. No retries permitted until 2026-07-16 17:33:53.250211362 +0000 UTC m=+10.236746959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7m98" (UniqueName: "kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98") pod "network-check-target-hr44p" (UID: "33b91431-5098-4f19-9e2f-489917539c2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:33:49.673058 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:49.673028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:49.673253 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.673154    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:33:49.673538 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:49.673518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:49.673644 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:49.673630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:49.673752 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.673731    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:33:49.673872 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:49.673852    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:33:51.668009 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:51.667945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:51.668440 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:51.667949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:51.668440 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:51.668088    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:51.668440 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:51.668082    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:33:51.668440 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:51.668227    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:33:51.668440 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:51.668325    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:33:53.183053 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:53.182448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:53.183053 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.182632    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:33:53.183053 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.182696    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs podName:c928af6c-6ba7-40bb-8e7a-4a418754b963 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:01.182677128 +0000 UTC m=+18.169212716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs") pod "network-metrics-daemon-gn4pn" (UID: "c928af6c-6ba7-40bb-8e7a-4a418754b963") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:33:53.283851 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:53.283809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:53.284042 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:53.283901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7m98\" (UniqueName: \"kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98\") pod \"network-check-target-hr44p\" (UID: \"33b91431-5098-4f19-9e2f-489917539c2f\") " pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:53.284109 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.284060    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:33:53.284109 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.284082    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:33:53.284109 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.284095    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7m98 for pod openshift-network-diagnostics/network-check-target-hr44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:33:53.284224 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.284154    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98 podName:33b91431-5098-4f19-9e2f-489917539c2f nodeName:}" failed. No retries permitted until 2026-07-16 17:34:01.284136596 +0000 UTC m=+18.270672194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7m98" (UniqueName: "kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98") pod "network-check-target-hr44p" (UID: "33b91431-5098-4f19-9e2f-489917539c2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:33:53.284646 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.284624    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:33:53.284758 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.284689    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret podName:ef3ddfba-d14b-493c-8c03-0c96efeb611b nodeName:}" failed. No retries permitted until 2026-07-16 17:34:01.284672985 +0000 UTC m=+18.271208569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret") pod "global-pull-secret-syncer-tnzn7" (UID: "ef3ddfba-d14b-493c-8c03-0c96efeb611b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:33:53.669273 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:53.669238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:53.669463 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.669345    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:33:53.669783 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:53.669757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:53.669924 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.669863    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:33:53.670017 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:53.670001    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:53.670106 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:53.670087    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:33:55.668297 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:55.668261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:55.668854 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:55.668404    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:33:55.668854 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:55.668271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:55.668854 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:55.668778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:55.669015 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:55.668869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:33:55.669015 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:55.668959    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:33:57.667864 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:57.667824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:57.668350 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:57.667961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:57.668350 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:57.667972    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:33:57.668350 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:57.668006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:57.668350 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:57.668112    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:33:57.668350 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:57.668197    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:33:59.670603 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:59.670574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:33:59.670603 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:59.670596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:33:59.670989 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:33:59.670575    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:33:59.670989 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:59.670684    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:33:59.670989 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:59.670737    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:33:59.670989 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:33:59.670838    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:34:01.246761 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:01.246717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:01.247251 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.246876    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:01.247251 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.246943    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs podName:c928af6c-6ba7-40bb-8e7a-4a418754b963 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:17.246923977 +0000 UTC m=+34.233459562 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs") pod "network-metrics-daemon-gn4pn" (UID: "c928af6c-6ba7-40bb-8e7a-4a418754b963") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:01.347172 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:01.347141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:01.347341 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:01.347213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7m98\" (UniqueName: \"kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98\") pod \"network-check-target-hr44p\" (UID: \"33b91431-5098-4f19-9e2f-489917539c2f\") " pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:01.347341 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.347311    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:01.347458 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.347342    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:01.347458 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.347363    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:01.347458 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.347376    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7m98 for pod openshift-network-diagnostics/network-check-target-hr44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:01.347458 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.347386    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret podName:ef3ddfba-d14b-493c-8c03-0c96efeb611b nodeName:}" failed. No retries permitted until 2026-07-16 17:34:17.347363943 +0000 UTC m=+34.333899530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret") pod "global-pull-secret-syncer-tnzn7" (UID: "ef3ddfba-d14b-493c-8c03-0c96efeb611b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:01.347458 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.347415    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98 podName:33b91431-5098-4f19-9e2f-489917539c2f nodeName:}" failed. No retries permitted until 2026-07-16 17:34:17.347403998 +0000 UTC m=+34.333939595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7m98" (UniqueName: "kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98") pod "network-check-target-hr44p" (UID: "33b91431-5098-4f19-9e2f-489917539c2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:01.667958 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:01.667874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:01.668113 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:01.667874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:01.668113 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.668001    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:34:01.668113 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:01.667880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:01.668113 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.668080    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:34:01.668294 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:01.668154    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:34:03.669181 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:03.669143    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:03.669657 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:03.669235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:03.669657 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:03.669255    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:34:03.669657 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:03.669359    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:34:03.669657 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:03.669396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:03.669657 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:03.669477    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:34:05.671287 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.671037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:05.671834 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.671073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:05.671834 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:05.671370    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:34:05.671834 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.671098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:05.671834 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:05.671438    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:34:05.671834 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:05.671506    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:34:05.769401 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.769381    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:34:05.769825 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.769798    2563 generic.go:356] "Generic (PLEG): container finished" podID="ffe1aeb1-9246-430c-964b-2eb6db9dd017" containerID="243d2704891f161db98bc6d130444010fb198d6b2b2376917b3b1a3a68e65ab9" exitCode=1
Jul 16 17:34:05.769938 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.769880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" event={"ID":"ffe1aeb1-9246-430c-964b-2eb6db9dd017","Type":"ContainerStarted","Data":"ae26c883b6a3601ce3d0f600a65021404dce56c5c1347cba5aa74fb809d6e475"}
Jul 16 17:34:05.770003 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.769950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" event={"ID":"ffe1aeb1-9246-430c-964b-2eb6db9dd017","Type":"ContainerStarted","Data":"9d53d96b03c9f8a84aae726b0f24e6db775453d5cc7a0f8f5a26266d83b2ad3a"}
Jul 16 17:34:05.770003 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.769996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" event={"ID":"ffe1aeb1-9246-430c-964b-2eb6db9dd017","Type":"ContainerStarted","Data":"796d40a91030cb3eaf3aa50387cafe3d1abf96393fdb03ed88028131373e01c0"}
Jul 16 17:34:05.770093 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.770029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" event={"ID":"ffe1aeb1-9246-430c-964b-2eb6db9dd017","Type":"ContainerDied","Data":"243d2704891f161db98bc6d130444010fb198d6b2b2376917b3b1a3a68e65ab9"}
Jul 16 17:34:05.770093 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.770074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" event={"ID":"ffe1aeb1-9246-430c-964b-2eb6db9dd017","Type":"ContainerStarted","Data":"ed772db30270d8940b56dfac3e7273ce78293ffd5af278d65146f924d6d6bf16"}
Jul 16 17:34:05.771599 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.771538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mxzqv" event={"ID":"fa1f394b-ac23-4640-86ed-75f04466fa1e","Type":"ContainerStarted","Data":"8637ff150d1518aa374a50ea1e8147c3192b9f66a5cc3146d1bc70883171571f"}
Jul 16 17:34:05.773407 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.773383    2563 generic.go:356] "Generic (PLEG): container finished" podID="dea026da-38bb-4bf7-9382-7405d0abc6d4" containerID="e1e8951b448f5c11808b5ffa702e6c1ff07e04f7f35401a938f4b0f8db9f81d7" exitCode=0
Jul 16 17:34:05.773513 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.773467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sdp" event={"ID":"dea026da-38bb-4bf7-9382-7405d0abc6d4","Type":"ContainerDied","Data":"e1e8951b448f5c11808b5ffa702e6c1ff07e04f7f35401a938f4b0f8db9f81d7"}
Jul 16 17:34:05.775649 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.775529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-949cg" event={"ID":"63d63b93-8c90-401b-ba2c-1585ea70fb35","Type":"ContainerStarted","Data":"2288196831d212cd4f1cb0c70fb01a529137e2d73a01997a7c12c59859366a55"}
Jul 16 17:34:05.777206 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.777144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-whxjd" event={"ID":"4a921cd4-90af-41b4-828f-9c74ce755d39","Type":"ContainerStarted","Data":"1eb61bf12d3639f3e604c55c10894a671f03ba3ab9f7c1885e4c55ba5312c06d"}
Jul 16 17:34:05.778988 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.778957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vdpc9" event={"ID":"04ee78ee-1c82-4394-8b56-23cda890ae80","Type":"ContainerStarted","Data":"b326f36de7ce4a9c126e7736caa83034a9930634870b786238b9393f95e60f14"}
Jul 16 17:34:05.780956 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.780910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rsfgv" event={"ID":"d656ee83-7adb-4ed3-8255-215c6736b17a","Type":"ContainerStarted","Data":"4418e87c5d77085b274bdd3c984419dd0e381699d2124820b0bafe79205f6539"}
Jul 16 17:34:05.782521 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.782500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt" event={"ID":"07d6c3ec-f524-4f27-98a6-2e0622dd3926","Type":"ContainerStarted","Data":"a43a651db698c4eea31480c4155e414dc791a6f72d5e66a98364c877bd0bb4e9"}
Jul 16 17:34:05.795147 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.795093    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-218.ec2.internal" podStartSLOduration=21.795075733 podStartE2EDuration="21.795075733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:48.757135815 +0000 UTC m=+5.743671451" watchObservedRunningTime="2026-07-16 17:34:05.795075733 +0000 UTC m=+22.781611339"
Jul 16 17:34:05.830620 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.830545    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mxzqv" podStartSLOduration=10.004155052 podStartE2EDuration="22.83052816s" totalImagesPullingTime="12.826373108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:05.804682531 +0000 UTC m=+22.791218136" watchObservedRunningTime="2026-07-16 17:34:05.83052816 +0000 UTC m=+22.817063810"
Jul 16 17:34:05.856918 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.856857    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-whxjd" podStartSLOduration=4.537454637 podStartE2EDuration="22.856834635s" totalImagesPullingTime="18.319379998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:05.830399061 +0000 UTC m=+22.816934669" watchObservedRunningTime="2026-07-16 17:34:05.856834635 +0000 UTC m=+22.843370244"
Jul 16 17:34:05.896679 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.896628    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-949cg" podStartSLOduration=4.636813388 podStartE2EDuration="22.896611345s" totalImagesPullingTime="18.259797957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:05.8588582 +0000 UTC m=+22.845393803" watchObservedRunningTime="2026-07-16 17:34:05.896611345 +0000 UTC m=+22.883146950"
Jul 16 17:34:05.934469 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.934425    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rsfgv" podStartSLOduration=4.666541903 podStartE2EDuration="22.934406841s" totalImagesPullingTime="18.267864938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:05.897728846 +0000 UTC m=+22.884264455" watchObservedRunningTime="2026-07-16 17:34:05.934406841 +0000 UTC m=+22.920942441"
Jul 16 17:34:05.935021 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:05.934995    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vdpc9" podStartSLOduration=4.668629808 podStartE2EDuration="22.93498503s" totalImagesPullingTime="18.266355222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:05.934541457 +0000 UTC m=+22.921077063" watchObservedRunningTime="2026-07-16 17:34:05.93498503 +0000 UTC m=+22.921520635"
Jul 16 17:34:06.197190 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.197133    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:34:06.197957 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.197937    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:34:06.204363 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.204339    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 17:34:06.546365 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.546248    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T17:34:06.204359657Z","UUID":"30cbb039-6763-4f2b-92da-48d97c5c82ab","Handler":null,"Name":"","Endpoint":""}
Jul 16 17:34:06.547924 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.547886    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 17:34:06.548073 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.547937    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 17:34:06.787698 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.787659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wnvg9" event={"ID":"2f7cb219-cff0-464c-b353-97537f8d29c3","Type":"ContainerStarted","Data":"797ac4afc4b9d2a22a9551d65efd4f553b5b8578bd1e370a9cb4fa370b3e6649"}
Jul 16 17:34:06.790228 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.790159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt" event={"ID":"07d6c3ec-f524-4f27-98a6-2e0622dd3926","Type":"ContainerStarted","Data":"dc8febb5e949e987a26fa564437b09c356b4e2614521243b6d2c1804eae1ec84"}
Jul 16 17:34:06.793728 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.793703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:34:06.794134 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.794109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" event={"ID":"ffe1aeb1-9246-430c-964b-2eb6db9dd017","Type":"ContainerStarted","Data":"194cf311a8ea6cbfc6e8b6fa129fd41c6690a709ecae6e2b0061fa2a886ef2b4"}
Jul 16 17:34:06.795070 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.795049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:34:06.795527 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.795513    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rsfgv"
Jul 16 17:34:06.839611 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:06.839512    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wnvg9" podStartSLOduration=11.0203726 podStartE2EDuration="23.839496849s" totalImagesPullingTime="12.819124249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:06.811613279 +0000 UTC m=+23.798148884" watchObservedRunningTime="2026-07-16 17:34:06.839496849 +0000 UTC m=+23.826032455"
Jul 16 17:34:07.672442 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:07.672411    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:07.672442 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:07.672431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:07.672700 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:07.672523    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:07.672700 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:07.672567    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:34:07.672700 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:07.672636    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:34:07.672815 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:07.672750    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:34:07.799112 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:07.799071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt" event={"ID":"07d6c3ec-f524-4f27-98a6-2e0622dd3926","Type":"ContainerStarted","Data":"299305e7a9e260dce2981ebc2a90ef7215da09d85bf55369411df3c64b7e14d8"}
Jul 16 17:34:08.805718 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:08.805567    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:34:08.806175 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:08.806118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" event={"ID":"ffe1aeb1-9246-430c-964b-2eb6db9dd017","Type":"ContainerStarted","Data":"0c9dea0a6e94f2bc00e5c8cab594a6f7b4c970fab8f8c858392e513c2e66fbc6"}
Jul 16 17:34:09.668701 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:09.668665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:09.668875 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:09.668776    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:34:09.668875 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:09.668665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:09.668875 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:09.668813    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:09.668983 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:09.668907    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:34:09.669031 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:09.669011    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:34:10.812158 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.811991    2563 generic.go:356] "Generic (PLEG): container finished" podID="dea026da-38bb-4bf7-9382-7405d0abc6d4" containerID="b17ef2da1317c53a4f86a50d760afb3679660d76c8bb51323d6a4dc49e337aed" exitCode=0
Jul 16 17:34:10.812939 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.812075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sdp" event={"ID":"dea026da-38bb-4bf7-9382-7405d0abc6d4","Type":"ContainerDied","Data":"b17ef2da1317c53a4f86a50d760afb3679660d76c8bb51323d6a4dc49e337aed"}
Jul 16 17:34:10.815686 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.815665    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:34:10.816022 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.815997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" event={"ID":"ffe1aeb1-9246-430c-964b-2eb6db9dd017","Type":"ContainerStarted","Data":"c120ead35c918fa83318d6e4f53edf1ceb6e48319f2d51b15d06242d18df614c"}
Jul 16 17:34:10.816448 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.816272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:34:10.816448 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.816343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:34:10.816448 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.816399    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:34:10.816636 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.816487    2563 scope.go:117] "RemoveContainer" containerID="243d2704891f161db98bc6d130444010fb198d6b2b2376917b3b1a3a68e65ab9"
Jul 16 17:34:10.833063 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.832569    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:34:10.835001 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.834980    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:34:10.863540 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:10.863489    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dglpt" podStartSLOduration=7.847174358 podStartE2EDuration="27.863465015s" totalImagesPullingTime="20.016290657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:07.828646782 +0000 UTC m=+24.815182383" watchObservedRunningTime="2026-07-16 17:34:10.863465015 +0000 UTC m=+27.850000615"
Jul 16 17:34:11.668747 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:11.668654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:11.668747 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:11.668657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:11.668965 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:11.668781    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:34:11.668965 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:11.668815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:11.669052 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:11.668965    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:34:11.669105 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:11.669067    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:34:11.822032 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:11.822001    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:34:11.822436 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:11.822366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" event={"ID":"ffe1aeb1-9246-430c-964b-2eb6db9dd017","Type":"ContainerStarted","Data":"4443db15010b0ac303047ed51a40385a77551942ef6e7d13f961a8234579522d"}
Jul 16 17:34:11.881203 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:11.881154    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts" podStartSLOduration=10.27491581 podStartE2EDuration="28.88113867s" totalImagesPullingTime="18.60622286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:11.879754925 +0000 UTC m=+28.866290555" watchObservedRunningTime="2026-07-16 17:34:11.88113867 +0000 UTC m=+28.867674345"
Jul 16 17:34:12.600152 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:12.600116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hr44p"]
Jul 16 17:34:12.600306 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:12.600236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:12.600350 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:12.600329    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:34:12.606711 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:12.606683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gn4pn"]
Jul 16 17:34:12.606839 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:12.606784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:12.606892 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:12.606875    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:34:12.619720 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:12.619649    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tnzn7"]
Jul 16 17:34:12.619839 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:12.619728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:12.619839 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:12.619809    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:34:12.826191 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:12.825961    2563 generic.go:356] "Generic (PLEG): container finished" podID="dea026da-38bb-4bf7-9382-7405d0abc6d4" containerID="036cb03ea51066546bff04b76f223d6464d4e153b0d4c032776dea5b7c5d14a1" exitCode=0
Jul 16 17:34:12.826611 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:12.826042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sdp" event={"ID":"dea026da-38bb-4bf7-9382-7405d0abc6d4","Type":"ContainerDied","Data":"036cb03ea51066546bff04b76f223d6464d4e153b0d4c032776dea5b7c5d14a1"}
Jul 16 17:34:14.668746 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:14.668712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:14.669461 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:14.668753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:14.669461 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:14.668753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:14.669461 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:14.668830    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:34:14.669461 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:14.668921    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:34:14.669461 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:14.668968    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:34:14.834004 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:14.833912    2563 generic.go:356] "Generic (PLEG): container finished" podID="dea026da-38bb-4bf7-9382-7405d0abc6d4" containerID="2663af9ff7fc586186c953b4acbdbc8003f88fd15a910d3947e479712df3fca5" exitCode=0
Jul 16 17:34:14.834004 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:14.833962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sdp" event={"ID":"dea026da-38bb-4bf7-9382-7405d0abc6d4","Type":"ContainerDied","Data":"2663af9ff7fc586186c953b4acbdbc8003f88fd15a910d3947e479712df3fca5"}
Jul 16 17:34:16.668542 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:16.668498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:16.668542 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:16.668540    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:16.669114 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:16.668546    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:16.669114 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:16.668652    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gn4pn" podUID="c928af6c-6ba7-40bb-8e7a-4a418754b963"
Jul 16 17:34:16.669114 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:16.668782    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tnzn7" podUID="ef3ddfba-d14b-493c-8c03-0c96efeb611b"
Jul 16 17:34:16.669114 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:16.668872    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr44p" podUID="33b91431-5098-4f19-9e2f-489917539c2f"
Jul 16 17:34:17.261437 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.261392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:17.261814 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:17.261507    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:17.261814 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:17.261584    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs podName:c928af6c-6ba7-40bb-8e7a-4a418754b963 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:49.261545803 +0000 UTC m=+66.248081387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs") pod "network-metrics-daemon-gn4pn" (UID: "c928af6c-6ba7-40bb-8e7a-4a418754b963") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:17.361942 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.361904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7m98\" (UniqueName: \"kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98\") pod \"network-check-target-hr44p\" (UID: \"33b91431-5098-4f19-9e2f-489917539c2f\") " pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:17.362092 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.361982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:17.362092 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:17.362078    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:17.362092 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:17.362091    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:17.362225 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:17.362103    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:17.362225 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:17.362116    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7m98 for pod openshift-network-diagnostics/network-check-target-hr44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:17.362225 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:17.362153    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret podName:ef3ddfba-d14b-493c-8c03-0c96efeb611b nodeName:}" failed. No retries permitted until 2026-07-16 17:34:49.362135523 +0000 UTC m=+66.348671125 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret") pod "global-pull-secret-syncer-tnzn7" (UID: "ef3ddfba-d14b-493c-8c03-0c96efeb611b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:17.362225 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:17.362172    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98 podName:33b91431-5098-4f19-9e2f-489917539c2f nodeName:}" failed. No retries permitted until 2026-07-16 17:34:49.362162428 +0000 UTC m=+66.348698015 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7m98" (UniqueName: "kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98") pod "network-check-target-hr44p" (UID: "33b91431-5098-4f19-9e2f-489917539c2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:17.801543 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.801510    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-218.ec2.internal" event="NodeReady"
Jul 16 17:34:17.802094 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.801668    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 17:34:17.860157 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.860065    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nqwmf"]
Jul 16 17:34:17.885816 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.885782    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6z8w2"]
Jul 16 17:34:17.885969 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.885928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:17.889164 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.889137    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 17:34:17.889309 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.889204    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 17:34:17.889381 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.889362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-24sj8\""
Jul 16 17:34:17.900888 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.900860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6z8w2"]
Jul 16 17:34:17.901030 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.900904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nqwmf"]
Jul 16 17:34:17.901030 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.900974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6z8w2"
Jul 16 17:34:17.904581 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.904349    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ndv45\""
Jul 16 17:34:17.904581 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.904418    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 17:34:17.905542 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.905521    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 17:34:17.905668 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:17.905600    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 17:34:18.067815 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.067777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.067815 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.067812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrbv\" (UniqueName: \"kubernetes.io/projected/b1836785-46a4-4231-a977-eee3a2d8b563-kube-api-access-dlrbv\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.068040 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.067853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd249ea5-c446-40d1-97b1-97223cc47d84-cert\") pod \"ingress-canary-6z8w2\" (UID: \"fd249ea5-c446-40d1-97b1-97223cc47d84\") " pod="openshift-ingress-canary/ingress-canary-6z8w2"
Jul 16 17:34:18.068040 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.067906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rm5v\" (UniqueName: \"kubernetes.io/projected/fd249ea5-c446-40d1-97b1-97223cc47d84-kube-api-access-7rm5v\") pod \"ingress-canary-6z8w2\" (UID: \"fd249ea5-c446-40d1-97b1-97223cc47d84\") " pod="openshift-ingress-canary/ingress-canary-6z8w2"
Jul 16 17:34:18.068040 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.067978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1836785-46a4-4231-a977-eee3a2d8b563-tmp-dir\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.068040 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.068002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1836785-46a4-4231-a977-eee3a2d8b563-config-volume\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.169183 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.169090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.169183 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.169136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrbv\" (UniqueName: \"kubernetes.io/projected/b1836785-46a4-4231-a977-eee3a2d8b563-kube-api-access-dlrbv\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.169414 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.169194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd249ea5-c446-40d1-97b1-97223cc47d84-cert\") pod \"ingress-canary-6z8w2\" (UID: \"fd249ea5-c446-40d1-97b1-97223cc47d84\") " pod="openshift-ingress-canary/ingress-canary-6z8w2"
Jul 16 17:34:18.169414 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.169217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rm5v\" (UniqueName: \"kubernetes.io/projected/fd249ea5-c446-40d1-97b1-97223cc47d84-kube-api-access-7rm5v\") pod \"ingress-canary-6z8w2\" (UID: \"fd249ea5-c446-40d1-97b1-97223cc47d84\") " pod="openshift-ingress-canary/ingress-canary-6z8w2"
Jul 16 17:34:18.169414 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.169265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1836785-46a4-4231-a977-eee3a2d8b563-tmp-dir\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.169414 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:18.169269    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:34:18.169414 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.169291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1836785-46a4-4231-a977-eee3a2d8b563-config-volume\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.169414 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:18.169348    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls podName:b1836785-46a4-4231-a977-eee3a2d8b563 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:18.669326567 +0000 UTC m=+35.655862171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls") pod "dns-default-nqwmf" (UID: "b1836785-46a4-4231-a977-eee3a2d8b563") : secret "dns-default-metrics-tls" not found
Jul 16 17:34:18.169754 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.169734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1836785-46a4-4231-a977-eee3a2d8b563-tmp-dir\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.170028 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.170003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1836785-46a4-4231-a977-eee3a2d8b563-config-volume\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.174774 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.174583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd249ea5-c446-40d1-97b1-97223cc47d84-cert\") pod \"ingress-canary-6z8w2\" (UID: \"fd249ea5-c446-40d1-97b1-97223cc47d84\") " pod="openshift-ingress-canary/ingress-canary-6z8w2"
Jul 16 17:34:18.186070 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.186037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrbv\" (UniqueName: \"kubernetes.io/projected/b1836785-46a4-4231-a977-eee3a2d8b563-kube-api-access-dlrbv\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.186465 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.186441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rm5v\" (UniqueName: \"kubernetes.io/projected/fd249ea5-c446-40d1-97b1-97223cc47d84-kube-api-access-7rm5v\") pod \"ingress-canary-6z8w2\" (UID: \"fd249ea5-c446-40d1-97b1-97223cc47d84\") " pod="openshift-ingress-canary/ingress-canary-6z8w2"
Jul 16 17:34:18.211073 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.211032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6z8w2"
Jul 16 17:34:18.668425 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.668389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:18.668653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.668388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:18.668722 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.668403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:18.672222 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.672189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xfj9z\""
Jul 16 17:34:18.673760 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.673723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:18.673900 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:18.673881    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:34:18.673984 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.673953    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 17:34:18.674045 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:18.674003    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls podName:b1836785-46a4-4231-a977-eee3a2d8b563 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:19.673982521 +0000 UTC m=+36.660518124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls") pod "dns-default-nqwmf" (UID: "b1836785-46a4-4231-a977-eee3a2d8b563") : secret "dns-default-metrics-tls" not found
Jul 16 17:34:18.674045 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.674003    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2qkbl\""
Jul 16 17:34:18.674045 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.674025    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 17:34:18.674196 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.673955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 17:34:18.674196 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:18.673897    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 17:34:19.043855 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:19.043818    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6z8w2"]
Jul 16 17:34:19.089439 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:34:19.089402    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd249ea5_c446_40d1_97b1_97223cc47d84.slice/crio-a3be2525587213aa284550c0cdd2c6ed0500d1f9dd619f7d9ffeb6979dad9dff WatchSource:0}: Error finding container a3be2525587213aa284550c0cdd2c6ed0500d1f9dd619f7d9ffeb6979dad9dff: Status 404 returned error can't find the container with id a3be2525587213aa284550c0cdd2c6ed0500d1f9dd619f7d9ffeb6979dad9dff
Jul 16 17:34:19.682990 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:19.682956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:19.683160 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:19.683123    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:34:19.683235 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:19.683200    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls podName:b1836785-46a4-4231-a977-eee3a2d8b563 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:21.683179768 +0000 UTC m=+38.669715368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls") pod "dns-default-nqwmf" (UID: "b1836785-46a4-4231-a977-eee3a2d8b563") : secret "dns-default-metrics-tls" not found
Jul 16 17:34:19.849387 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:19.849346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6z8w2" event={"ID":"fd249ea5-c446-40d1-97b1-97223cc47d84","Type":"ContainerStarted","Data":"a3be2525587213aa284550c0cdd2c6ed0500d1f9dd619f7d9ffeb6979dad9dff"}
Jul 16 17:34:19.852798 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:19.852764    2563 generic.go:356] "Generic (PLEG): container finished" podID="dea026da-38bb-4bf7-9382-7405d0abc6d4" containerID="0a84bf95e656214f04198404830f1206e4a2fa6bd4b20b1538ba1287c07c73e6" exitCode=0
Jul 16 17:34:19.853022 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:19.852825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sdp" event={"ID":"dea026da-38bb-4bf7-9382-7405d0abc6d4","Type":"ContainerDied","Data":"0a84bf95e656214f04198404830f1206e4a2fa6bd4b20b1538ba1287c07c73e6"}
Jul 16 17:34:20.859883 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:20.859844    2563 generic.go:356] "Generic (PLEG): container finished" podID="dea026da-38bb-4bf7-9382-7405d0abc6d4" containerID="835de4b15aa12ea0c428b1d8da68a0cf9a518b7c2537c2d5a0d9d533593997ec" exitCode=0
Jul 16 17:34:20.860325 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:20.859885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sdp" event={"ID":"dea026da-38bb-4bf7-9382-7405d0abc6d4","Type":"ContainerDied","Data":"835de4b15aa12ea0c428b1d8da68a0cf9a518b7c2537c2d5a0d9d533593997ec"}
Jul 16 17:34:21.700339 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:21.700096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:21.700511 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:21.700244    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:34:21.700511 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:21.700483    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls podName:b1836785-46a4-4231-a977-eee3a2d8b563 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:25.700463812 +0000 UTC m=+42.686999401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls") pod "dns-default-nqwmf" (UID: "b1836785-46a4-4231-a977-eee3a2d8b563") : secret "dns-default-metrics-tls" not found
Jul 16 17:34:21.869359 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:21.869323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sdp" event={"ID":"dea026da-38bb-4bf7-9382-7405d0abc6d4","Type":"ContainerStarted","Data":"4ac4d8877ac094925bcc69684567624e1695971c7ab9ccf6c27b50f93820627b"}
Jul 16 17:34:21.870874 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:21.870845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6z8w2" event={"ID":"fd249ea5-c446-40d1-97b1-97223cc47d84","Type":"ContainerStarted","Data":"1274307cb198dd6cde5ce1d087ec2f0c95d96f21f3b5c6aab5a26fcb05d55081"}
Jul 16 17:34:21.936948 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:21.936889    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-25sdp" podStartSLOduration=10.045787358 podStartE2EDuration="38.936872451s" totalImagesPullingTime="28.891085093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:21.934691047 +0000 UTC m=+38.921226654" watchObservedRunningTime="2026-07-16 17:34:21.936872451 +0000 UTC m=+38.923408056"
Jul 16 17:34:21.988133 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:21.988084    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6z8w2" podStartSLOduration=2.7110350199999997 podStartE2EDuration="4.988065726s" totalImagesPullingTime="2.277030706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:21.986416932 +0000 UTC m=+38.972952537" watchObservedRunningTime="2026-07-16 17:34:21.988065726 +0000 UTC m=+38.974601362"
Jul 16 17:34:25.729292 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:25.729251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:25.729853 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:25.729396    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:34:25.729853 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:25.729461    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls podName:b1836785-46a4-4231-a977-eee3a2d8b563 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:33.729446011 +0000 UTC m=+50.715981594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls") pod "dns-default-nqwmf" (UID: "b1836785-46a4-4231-a977-eee3a2d8b563") : secret "dns-default-metrics-tls" not found
Jul 16 17:34:33.785615 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:33.785571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:33.786063 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:33.785698    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:34:33.786063 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:33.785761    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls podName:b1836785-46a4-4231-a977-eee3a2d8b563 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:49.785747489 +0000 UTC m=+66.772283073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls") pod "dns-default-nqwmf" (UID: "b1836785-46a4-4231-a977-eee3a2d8b563") : secret "dns-default-metrics-tls" not found
Jul 16 17:34:42.846088 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:42.846059    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4ts"
Jul 16 17:34:49.298035 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.297989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:34:49.301440 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.301420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 17:34:49.308600 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:49.308569    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 17:34:49.308702 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:49.308691    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs podName:c928af6c-6ba7-40bb-8e7a-4a418754b963 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:53.308674608 +0000 UTC m=+130.295210195 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs") pod "network-metrics-daemon-gn4pn" (UID: "c928af6c-6ba7-40bb-8e7a-4a418754b963") : secret "metrics-daemon-secret" not found
Jul 16 17:34:49.399083 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.399037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7m98\" (UniqueName: \"kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98\") pod \"network-check-target-hr44p\" (UID: \"33b91431-5098-4f19-9e2f-489917539c2f\") " pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:49.399254 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.399100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:49.401881 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.401859    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 17:34:49.401950 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.401875    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 17:34:49.412810 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.412783    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 17:34:49.413658 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.413631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef3ddfba-d14b-493c-8c03-0c96efeb611b-original-pull-secret\") pod \"global-pull-secret-syncer-tnzn7\" (UID: \"ef3ddfba-d14b-493c-8c03-0c96efeb611b\") " pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:49.422525 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.422495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7m98\" (UniqueName: \"kubernetes.io/projected/33b91431-5098-4f19-9e2f-489917539c2f-kube-api-access-d7m98\") pod \"network-check-target-hr44p\" (UID: \"33b91431-5098-4f19-9e2f-489917539c2f\") " pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:49.580822 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.580729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tnzn7"
Jul 16 17:34:49.590850 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.590828    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xfj9z\""
Jul 16 17:34:49.598483 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.598449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:49.768242 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.768211    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tnzn7"]
Jul 16 17:34:49.772184 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:34:49.772157    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3ddfba_d14b_493c_8c03_0c96efeb611b.slice/crio-eb79429a11386919a191a02acb650f0bd56de26b65753e60d37c8e41638ebdf5 WatchSource:0}: Error finding container eb79429a11386919a191a02acb650f0bd56de26b65753e60d37c8e41638ebdf5: Status 404 returned error can't find the container with id eb79429a11386919a191a02acb650f0bd56de26b65753e60d37c8e41638ebdf5
Jul 16 17:34:49.791396 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.791364    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hr44p"]
Jul 16 17:34:49.794668 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:34:49.794633    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b91431_5098_4f19_9e2f_489917539c2f.slice/crio-2a6dedd857a07185da19c071859e3303aab9a8184e973e65332e329ad33bf024 WatchSource:0}: Error finding container 2a6dedd857a07185da19c071859e3303aab9a8184e973e65332e329ad33bf024: Status 404 returned error can't find the container with id 2a6dedd857a07185da19c071859e3303aab9a8184e973e65332e329ad33bf024
Jul 16 17:34:49.801320 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.801294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:34:49.801430 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:49.801413    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:34:49.801473 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:34:49.801462    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls podName:b1836785-46a4-4231-a977-eee3a2d8b563 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:21.801448589 +0000 UTC m=+98.787984174 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls") pod "dns-default-nqwmf" (UID: "b1836785-46a4-4231-a977-eee3a2d8b563") : secret "dns-default-metrics-tls" not found
Jul 16 17:34:49.935403 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.935313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hr44p" event={"ID":"33b91431-5098-4f19-9e2f-489917539c2f","Type":"ContainerStarted","Data":"2a6dedd857a07185da19c071859e3303aab9a8184e973e65332e329ad33bf024"}
Jul 16 17:34:49.936315 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:49.936295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tnzn7" event={"ID":"ef3ddfba-d14b-493c-8c03-0c96efeb611b","Type":"ContainerStarted","Data":"eb79429a11386919a191a02acb650f0bd56de26b65753e60d37c8e41638ebdf5"}
Jul 16 17:34:52.947357 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:52.947314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hr44p" event={"ID":"33b91431-5098-4f19-9e2f-489917539c2f","Type":"ContainerStarted","Data":"7fae3c1a567575c93074aa80e2d53137e6f3c203db20fc04cdb2e5c6e3b029ef"}
Jul 16 17:34:52.947796 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:52.947450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:34:52.966829 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:52.966769    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hr44p" podStartSLOduration=66.90627467 podStartE2EDuration="1m9.966747727s" totalImagesPullingTime="3.060473057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:52.966072398 +0000 UTC m=+69.952608004" watchObservedRunningTime="2026-07-16 17:34:52.966747727 +0000 UTC m=+69.953283336"
Jul 16 17:34:54.953975 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:54.953939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tnzn7" event={"ID":"ef3ddfba-d14b-493c-8c03-0c96efeb611b","Type":"ContainerStarted","Data":"c2e0b2bf45448dfc862d3b4181ff5679687bbc2c5ea8e364cfc7d8586b048e2e"}
Jul 16 17:34:54.975619 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:34:54.975527    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tnzn7" podStartSLOduration=67.462557534 podStartE2EDuration="1m11.97550584s" totalImagesPullingTime="4.512948306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:54.974439772 +0000 UTC m=+71.960975380" watchObservedRunningTime="2026-07-16 17:34:54.97550584 +0000 UTC m=+71.962041446"
Jul 16 17:35:21.825172 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.825029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:35:21.825681 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:21.825226    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:21.825681 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:21.825336    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls podName:b1836785-46a4-4231-a977-eee3a2d8b563 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:25.82531617 +0000 UTC m=+162.811851771 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls") pod "dns-default-nqwmf" (UID: "b1836785-46a4-4231-a977-eee3a2d8b563") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:21.943115 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.943077    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5cb66c8d8-nn55n"]
Jul 16 17:35:21.947318 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.947291    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"]
Jul 16 17:35:21.947464 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.947407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:21.949925 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.949903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:21.951881 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.951857    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 17:35:21.952504 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.952481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lx67b\""
Jul 16 17:35:21.952609 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.952535    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 16 17:35:21.952609 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.952573    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 17:35:21.952853 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.952833    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 16 17:35:21.954628 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.954610    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 16 17:35:21.955121 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.955106    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 17:35:21.955174 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.955117    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 16 17:35:21.957396 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.957376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dzqkw\""
Jul 16 17:35:21.957651 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.957636    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 16 17:35:21.961144 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.961125    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 17:35:21.963899 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.963884    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 16 17:35:21.972683 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.972657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cb66c8d8-nn55n"]
Jul 16 17:35:21.973240 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:21.973220    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"]
Jul 16 17:35:22.030514 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.030475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.030514 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.030516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wwn6\" (UniqueName: \"kubernetes.io/projected/54fdfddd-0dd4-4af2-90bc-ccaac4996484-kube-api-access-6wwn6\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:22.030790 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.030538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54fdfddd-0dd4-4af2-90bc-ccaac4996484-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:22.030790 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.030603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbshd\" (UniqueName: \"kubernetes.io/projected/5569cfbb-8a85-4f59-b995-32f0a18d7857-kube-api-access-gbshd\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.030790 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.030641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-stats-auth\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.030790 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.030658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.030790 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.030737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-default-certificate\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.030790 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.030776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:22.073932 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.073893    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"]
Jul 16 17:35:22.076914 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.076868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:22.079657 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.079635    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 16 17:35:22.079807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.079786    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lrndd\""
Jul 16 17:35:22.079917 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.079901    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 16 17:35:22.083520 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.083501    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:35:22.090024 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.089997    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"]
Jul 16 17:35:22.131495 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.131460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-default-certificate\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.131495 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.131498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:22.131737 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.131538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.131737 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.131596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wwn6\" (UniqueName: \"kubernetes.io/projected/54fdfddd-0dd4-4af2-90bc-ccaac4996484-kube-api-access-6wwn6\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:22.131737 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.131621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54fdfddd-0dd4-4af2-90bc-ccaac4996484-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:22.131737 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.131639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbshd\" (UniqueName: \"kubernetes.io/projected/5569cfbb-8a85-4f59-b995-32f0a18d7857-kube-api-access-gbshd\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.131737 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.131652    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:22.131737 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.131702    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.631681622 +0000 UTC m=+99.618217223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:22.131737 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.131658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-stats-auth\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.131737 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.131726    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls podName:54fdfddd-0dd4-4af2-90bc-ccaac4996484 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.631715882 +0000 UTC m=+99.618251466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v4b27" (UID: "54fdfddd-0dd4-4af2-90bc-ccaac4996484") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:22.132118 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.131756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.132118 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.131833    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:22.132118 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.131868    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.63185731 +0000 UTC m=+99.618392911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : secret "router-metrics-certs-default" not found
Jul 16 17:35:22.132479 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.132453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54fdfddd-0dd4-4af2-90bc-ccaac4996484-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:22.134127 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.134108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-default-certificate\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.134235 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.134161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-stats-auth\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.142387 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.142359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbshd\" (UniqueName: \"kubernetes.io/projected/5569cfbb-8a85-4f59-b995-32f0a18d7857-kube-api-access-gbshd\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.143325 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.143300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wwn6\" (UniqueName: \"kubernetes.io/projected/54fdfddd-0dd4-4af2-90bc-ccaac4996484-kube-api-access-6wwn6\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:22.172732 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.172697    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-6dgms"]
Jul 16 17:35:22.175687 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.175668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.178333 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.178303    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 16 17:35:22.178575 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.178542    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:35:22.178690 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.178674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 16 17:35:22.178758 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.178742    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 16 17:35:22.179266 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.179252    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gq4r2\""
Jul 16 17:35:22.184042 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.184018    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 16 17:35:22.198168 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.198139    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-6dgms"]
Jul 16 17:35:22.233003 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.232954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgn8n\" (UniqueName: \"kubernetes.io/projected/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-kube-api-access-pgn8n\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:22.233178 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.233099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:22.334212 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.334115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-trusted-ca\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.334212 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.334159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-serving-cert\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.334212 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.334178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8ln\" (UniqueName: \"kubernetes.io/projected/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-kube-api-access-tj8ln\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.334436 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.334316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:22.334436 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.334348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-config\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.334436 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.334377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgn8n\" (UniqueName: \"kubernetes.io/projected/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-kube-api-access-pgn8n\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:22.334538 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.334461    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 17:35:22.334538 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.334533    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls podName:0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.834513447 +0000 UTC m=+99.821049042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls") pod "cluster-samples-operator-65db858894-qb9dl" (UID: "0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb") : secret "samples-operator-tls" not found
Jul 16 17:35:22.345850 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.345818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgn8n\" (UniqueName: \"kubernetes.io/projected/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-kube-api-access-pgn8n\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:22.435032 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.434989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-trusted-ca\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.435032 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.435027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-serving-cert\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.435032 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.435044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8ln\" (UniqueName: \"kubernetes.io/projected/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-kube-api-access-tj8ln\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.435286 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.435142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-config\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.435897 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.435875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-config\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.435997 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.435979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-trusted-ca\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.437428 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.437411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-serving-cert\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.444472 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.444440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8ln\" (UniqueName: \"kubernetes.io/projected/b2c17e8e-e654-40b4-84a6-6685fcd6e3cb-kube-api-access-tj8ln\") pod \"console-operator-7866497494-6dgms\" (UID: \"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb\") " pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.485521 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.485485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:22.612718 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.612658    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-6dgms"]
Jul 16 17:35:22.615055 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:35:22.615031    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c17e8e_e654_40b4_84a6_6685fcd6e3cb.slice/crio-f9c3ef6623361a570fb2c547aa469bd082b838e017829bb71dfcbf43cde1c828 WatchSource:0}: Error finding container f9c3ef6623361a570fb2c547aa469bd082b838e017829bb71dfcbf43cde1c828: Status 404 returned error can't find the container with id f9c3ef6623361a570fb2c547aa469bd082b838e017829bb71dfcbf43cde1c828
Jul 16 17:35:22.637864 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.637826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.638032 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.637887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:22.638032 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.637934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:22.638032 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.638002    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.63798355 +0000 UTC m=+100.624519156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:22.638161 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.638052    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:22.638161 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.638055    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:22.638161 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.638097    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls podName:54fdfddd-0dd4-4af2-90bc-ccaac4996484 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.638086 +0000 UTC m=+100.624621587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v4b27" (UID: "54fdfddd-0dd4-4af2-90bc-ccaac4996484") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:22.638161 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.638114    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.638105384 +0000 UTC m=+100.624640967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : secret "router-metrics-certs-default" not found
Jul 16 17:35:22.839373 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:22.839328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:22.839753 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.839508    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 17:35:22.839753 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:22.839603    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls podName:0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.839586017 +0000 UTC m=+100.826121663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls") pod "cluster-samples-operator-65db858894-qb9dl" (UID: "0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb") : secret "samples-operator-tls" not found
Jul 16 17:35:23.024671 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:23.024640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-6dgms" event={"ID":"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb","Type":"ContainerStarted","Data":"f9c3ef6623361a570fb2c547aa469bd082b838e017829bb71dfcbf43cde1c828"}
Jul 16 17:35:23.646746 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:23.646705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:23.646939 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:23.646795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:23.646939 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:23.646851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:23.646939 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:23.646875    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:23.646939 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:23.646938    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls podName:54fdfddd-0dd4-4af2-90bc-ccaac4996484 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:25.646922829 +0000 UTC m=+102.633458413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v4b27" (UID: "54fdfddd-0dd4-4af2-90bc-ccaac4996484") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:23.647126 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:23.646968    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:23.647126 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:23.646972    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:25.646958023 +0000 UTC m=+102.633493608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:23.647126 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:23.647026    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:25.647011121 +0000 UTC m=+102.633546710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : secret "router-metrics-certs-default" not found
Jul 16 17:35:23.848919 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:23.848880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:23.849286 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:23.849055    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 17:35:23.849286 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:23.849148    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls podName:0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb nodeName:}" failed. No retries permitted until 2026-07-16 17:35:25.849124262 +0000 UTC m=+102.835659858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls") pod "cluster-samples-operator-65db858894-qb9dl" (UID: "0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb") : secret "samples-operator-tls" not found
Jul 16 17:35:23.952211 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:23.952127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hr44p"
Jul 16 17:35:25.031849 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.031821    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/0.log"
Jul 16 17:35:25.032218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.031861    2563 generic.go:356] "Generic (PLEG): container finished" podID="b2c17e8e-e654-40b4-84a6-6685fcd6e3cb" containerID="620f4b905f942e32126a8f01b5a814881b5804052a925de079631210fce676e3" exitCode=255
Jul 16 17:35:25.032218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.031894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-6dgms" event={"ID":"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb","Type":"ContainerDied","Data":"620f4b905f942e32126a8f01b5a814881b5804052a925de079631210fce676e3"}
Jul 16 17:35:25.032218 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.032131    2563 scope.go:117] "RemoveContainer" containerID="620f4b905f942e32126a8f01b5a814881b5804052a925de079631210fce676e3"
Jul 16 17:35:25.664656 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.664619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:25.664795 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.664669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:25.664795 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.664712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:25.664795 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:25.664776    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:25.664906 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:25.664815    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:29.664801879 +0000 UTC m=+106.651337463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:25.664906 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:25.664821    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:25.664906 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:25.664840    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:29.664822775 +0000 UTC m=+106.651358380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : secret "router-metrics-certs-default" not found
Jul 16 17:35:25.664906 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:25.664860    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls podName:54fdfddd-0dd4-4af2-90bc-ccaac4996484 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:29.664849164 +0000 UTC m=+106.651384754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v4b27" (UID: "54fdfddd-0dd4-4af2-90bc-ccaac4996484") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:25.823108 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.823070    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7qwdm"]
Jul 16 17:35:25.826134 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.826109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:25.829792 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.829755    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 17:35:25.829947 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.829926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 17:35:25.830211 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.830193    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 17:35:25.831103 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.831083    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 17:35:25.831198 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.831094    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nsb9l\""
Jul 16 17:35:25.844649 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.844612    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qwdm"]
Jul 16 17:35:25.866725 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.866687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:25.866950 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:25.866818    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 17:35:25.866950 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:25.866870    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls podName:0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb nodeName:}" failed. No retries permitted until 2026-07-16 17:35:29.866855918 +0000 UTC m=+106.853391502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls") pod "cluster-samples-operator-65db858894-qb9dl" (UID: "0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb") : secret "samples-operator-tls" not found
Jul 16 17:35:25.967254 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.967223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/80ceb78a-1472-4ff4-889f-dc7a16520c1c-crio-socket\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:25.967352 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.967274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/80ceb78a-1472-4ff4-889f-dc7a16520c1c-data-volume\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:25.967352 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.967289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:25.967442 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.967345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:25.967442 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:25.967387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr49c\" (UniqueName: \"kubernetes.io/projected/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-api-access-xr49c\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.036223 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.036191    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:35:26.036625 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.036616    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/0.log"
Jul 16 17:35:26.036674 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.036648    2563 generic.go:356] "Generic (PLEG): container finished" podID="b2c17e8e-e654-40b4-84a6-6685fcd6e3cb" containerID="7efeb1c1fcec481bd02910889a54af8bde41049f7329a1d8ef581a901aea69db" exitCode=255
Jul 16 17:35:26.036723 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.036679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-6dgms" event={"ID":"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb","Type":"ContainerDied","Data":"7efeb1c1fcec481bd02910889a54af8bde41049f7329a1d8ef581a901aea69db"}
Jul 16 17:35:26.036765 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.036723    2563 scope.go:117] "RemoveContainer" containerID="620f4b905f942e32126a8f01b5a814881b5804052a925de079631210fce676e3"
Jul 16 17:35:26.036966 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.036945    2563 scope.go:117] "RemoveContainer" containerID="7efeb1c1fcec481bd02910889a54af8bde41049f7329a1d8ef581a901aea69db"
Jul 16 17:35:26.037144 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:26.037126    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-6dgms_openshift-console-operator(b2c17e8e-e654-40b4-84a6-6685fcd6e3cb)\"" pod="openshift-console-operator/console-operator-7866497494-6dgms" podUID="b2c17e8e-e654-40b4-84a6-6685fcd6e3cb"
Jul 16 17:35:26.067854 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.067816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/80ceb78a-1472-4ff4-889f-dc7a16520c1c-crio-socket\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.068033 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.067878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/80ceb78a-1472-4ff4-889f-dc7a16520c1c-data-volume\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.068033 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.067895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.068033 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.067917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.068033 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.067938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr49c\" (UniqueName: \"kubernetes.io/projected/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-api-access-xr49c\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.068033 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.067945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/80ceb78a-1472-4ff4-889f-dc7a16520c1c-crio-socket\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.068259 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:26.068032    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:35:26.068259 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:26.068083    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:26.068259 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:26.068099    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:26.568082163 +0000 UTC m=+103.554617759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : configmap "kube-rbac-proxy" not found
Jul 16 17:35:26.068259 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:26.068144    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:26.568125584 +0000 UTC m=+103.554661175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:26.068259 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.068167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/80ceb78a-1472-4ff4-889f-dc7a16520c1c-data-volume\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.077884 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.077857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr49c\" (UniqueName: \"kubernetes.io/projected/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-api-access-xr49c\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.573818 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.573781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.573818 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:26.573826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:26.574044 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:26.573906    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:35:26.574044 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:26.573923    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:26.574044 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:26.573969    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:27.573955123 +0000 UTC m=+104.560490733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:26.574044 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:26.573985    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:27.573975269 +0000 UTC m=+104.560510853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : configmap "kube-rbac-proxy" not found
Jul 16 17:35:27.040646 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:27.040619    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:35:27.041031 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:27.040967    2563 scope.go:117] "RemoveContainer" containerID="7efeb1c1fcec481bd02910889a54af8bde41049f7329a1d8ef581a901aea69db"
Jul 16 17:35:27.041123 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:27.041103    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-6dgms_openshift-console-operator(b2c17e8e-e654-40b4-84a6-6685fcd6e3cb)\"" pod="openshift-console-operator/console-operator-7866497494-6dgms" podUID="b2c17e8e-e654-40b4-84a6-6685fcd6e3cb"
Jul 16 17:35:27.582359 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:27.582319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:27.582359 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:27.582368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:27.582616 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:27.582451    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:35:27.582616 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:27.582472    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:27.582616 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:27.582518    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:29.58250362 +0000 UTC m=+106.569039204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : configmap "kube-rbac-proxy" not found
Jul 16 17:35:27.582616 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:27.582531    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:29.58252555 +0000 UTC m=+106.569061134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:27.977812 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:27.977784    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mxzqv_fa1f394b-ac23-4640-86ed-75f04466fa1e/dns-node-resolver/0.log"
Jul 16 17:35:28.963106 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:28.963076    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vdpc9_04ee78ee-1c82-4394-8b56-23cda890ae80/node-ca/0.log"
Jul 16 17:35:29.562612 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:29.562580    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6z8w2_fd249ea5-c446-40d1-97b1-97223cc47d84/serve-healthcheck-canary/0.log"
Jul 16 17:35:29.600220 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:29.600176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:29.600374 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:29.600227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:29.600374 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.600311    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:35:29.600374 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.600322    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:29.600487 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.600397    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:33.600366979 +0000 UTC m=+110.586902582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : configmap "kube-rbac-proxy" not found
Jul 16 17:35:29.600487 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.600424    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:33.600414899 +0000 UTC m=+110.586950486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:29.701638 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:29.701598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:29.701801 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:29.701650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:29.701801 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:29.701697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:29.701801 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.701744    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:29.701921 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.701806    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:37.701791367 +0000 UTC m=+114.688326954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : secret "router-metrics-certs-default" not found
Jul 16 17:35:29.701921 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.701820    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:29.701921 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.701822    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:37.70181598 +0000 UTC m=+114.688351566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:29.701921 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.701871    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls podName:54fdfddd-0dd4-4af2-90bc-ccaac4996484 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:37.701858367 +0000 UTC m=+114.688393951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v4b27" (UID: "54fdfddd-0dd4-4af2-90bc-ccaac4996484") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:29.903710 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:29.903622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:29.903858 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.903771    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 17:35:29.903858 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:29.903835    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls podName:0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb nodeName:}" failed. No retries permitted until 2026-07-16 17:35:37.903819573 +0000 UTC m=+114.890355158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls") pod "cluster-samples-operator-65db858894-qb9dl" (UID: "0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb") : secret "samples-operator-tls" not found
Jul 16 17:35:32.485956 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:32.485916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:32.486443 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:32.485991    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:32.486443 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:32.486432    2563 scope.go:117] "RemoveContainer" containerID="7efeb1c1fcec481bd02910889a54af8bde41049f7329a1d8ef581a901aea69db"
Jul 16 17:35:32.486661 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:32.486636    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-6dgms_openshift-console-operator(b2c17e8e-e654-40b4-84a6-6685fcd6e3cb)\"" pod="openshift-console-operator/console-operator-7866497494-6dgms" podUID="b2c17e8e-e654-40b4-84a6-6685fcd6e3cb"
Jul 16 17:35:33.634754 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:33.634703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:33.635152 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:33.634777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:33.635152 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:33.634787    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:35:33.635152 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:33.634852    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:41.634834324 +0000 UTC m=+118.621369926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : configmap "kube-rbac-proxy" not found
Jul 16 17:35:33.635152 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:33.634912    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:33.635152 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:33.634972    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:41.634956967 +0000 UTC m=+118.621492551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:37.769321 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:37.769279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:37.769845 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:37.769341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:37.769845 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:37.769433    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:37.769845 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:37.769506    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls podName:54fdfddd-0dd4-4af2-90bc-ccaac4996484 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:53.769489391 +0000 UTC m=+130.756024997 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v4b27" (UID: "54fdfddd-0dd4-4af2-90bc-ccaac4996484") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:37.769845 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:37.769521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:37.769845 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:37.769638    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle podName:5569cfbb-8a85-4f59-b995-32f0a18d7857 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:53.769627343 +0000 UTC m=+130.756162952 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle") pod "router-default-5cb66c8d8-nn55n" (UID: "5569cfbb-8a85-4f59-b995-32f0a18d7857") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:37.771837 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:37.771811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5569cfbb-8a85-4f59-b995-32f0a18d7857-metrics-certs\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:37.971935 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:37.971888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:37.974283 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:37.974255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-qb9dl\" (UID: \"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:37.985292 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:37.985255    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"
Jul 16 17:35:38.120088 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:38.120055    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl"]
Jul 16 17:35:39.071851 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:39.071820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl" event={"ID":"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb","Type":"ContainerStarted","Data":"98a943697bb70467d7b09952457ff36803045dc5dca6cfc1682394d670a16841"}
Jul 16 17:35:41.079206 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:41.079168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl" event={"ID":"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb","Type":"ContainerStarted","Data":"6a0848676fdb4540e495013b17b6a50aa9543a95e04298a39d11a74b66a857ba"}
Jul 16 17:35:41.079724 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:41.079220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl" event={"ID":"0bca2ff6-03ae-4b5d-a01b-f2b58b9776fb","Type":"ContainerStarted","Data":"03a96b3997a61760ef94615d6eeab0e379f3a5c3fe9e1ce8f8ba7ce1c3dc6e22"}
Jul 16 17:35:41.099442 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:41.099382    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-qb9dl" podStartSLOduration=16.638043358 podStartE2EDuration="19.099364572s" totalImagesPullingTime="2.461321214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:41.09865044 +0000 UTC m=+118.085186046" watchObservedRunningTime="2026-07-16 17:35:41.099364572 +0000 UTC m=+118.085900177"
Jul 16 17:35:41.701920 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:41.701879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:41.701920 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:41.701927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:41.702147 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:41.702077    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:41.702197 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:41.702147    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls podName:80ceb78a-1472-4ff4-889f-dc7a16520c1c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:57.702127005 +0000 UTC m=+134.688662590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qwdm" (UID: "80ceb78a-1472-4ff4-889f-dc7a16520c1c") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:41.702573 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:41.702524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80ceb78a-1472-4ff4-889f-dc7a16520c1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:44.668529 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:44.668490    2563 scope.go:117] "RemoveContainer" containerID="7efeb1c1fcec481bd02910889a54af8bde41049f7329a1d8ef581a901aea69db"
Jul 16 17:35:45.093474 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:45.093439    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:35:45.093656 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:45.093496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-6dgms" event={"ID":"b2c17e8e-e654-40b4-84a6-6685fcd6e3cb","Type":"ContainerStarted","Data":"068aff90afdeb5fbad663f325cdf68639d66bcffbff8d6a99da7f548f60ec144"}
Jul 16 17:35:45.093838 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:45.093818    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:45.122828 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:45.122763    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-6dgms" podStartSLOduration=21.361381118 podStartE2EDuration="23.122741599s" totalImagesPullingTime="1.761360481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:45.121909463 +0000 UTC m=+122.108445070" watchObservedRunningTime="2026-07-16 17:35:45.122741599 +0000 UTC m=+122.109277206"
Jul 16 17:35:45.369683 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:45.369540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-6dgms"
Jul 16 17:35:52.598566 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.598525    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"]
Jul 16 17:35:52.602598 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.601890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"
Jul 16 17:35:52.606448 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.606418    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 17:35:52.606627 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.606447    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 17:35:52.607391 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.607377    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 17:35:52.608703 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.608673    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xv79d\""
Jul 16 17:35:52.608824 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.608705    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 16 17:35:52.618254 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.618225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"]
Jul 16 17:35:52.690077 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.690046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15ca0e4b-940b-48b3-8723-5bdf9de14ac3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75f94864-9z6dm\" (UID: \"15ca0e4b-940b-48b3-8723-5bdf9de14ac3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"
Jul 16 17:35:52.690077 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.690080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26k58\" (UniqueName: \"kubernetes.io/projected/15ca0e4b-940b-48b3-8723-5bdf9de14ac3-kube-api-access-26k58\") pod \"managed-serviceaccount-addon-agent-75f94864-9z6dm\" (UID: \"15ca0e4b-940b-48b3-8723-5bdf9de14ac3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"
Jul 16 17:35:52.697506 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.697475    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-cr7t9"]
Jul 16 17:35:52.699716 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.699695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-cr7t9"
Jul 16 17:35:52.703065 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.703039    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-cfmrb\""
Jul 16 17:35:52.703690 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.703671    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 17:35:52.703962 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.703941    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 17:35:52.713745 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.713719    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-cr7t9"]
Jul 16 17:35:52.791342 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.791307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15ca0e4b-940b-48b3-8723-5bdf9de14ac3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75f94864-9z6dm\" (UID: \"15ca0e4b-940b-48b3-8723-5bdf9de14ac3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"
Jul 16 17:35:52.791342 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.791345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26k58\" (UniqueName: \"kubernetes.io/projected/15ca0e4b-940b-48b3-8723-5bdf9de14ac3-kube-api-access-26k58\") pod \"managed-serviceaccount-addon-agent-75f94864-9z6dm\" (UID: \"15ca0e4b-940b-48b3-8723-5bdf9de14ac3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"
Jul 16 17:35:52.791590 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.791496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b2rd\" (UniqueName: \"kubernetes.io/projected/bf23d9f0-60cd-49e2-b943-098589373c45-kube-api-access-5b2rd\") pod \"downloads-68c9cbb497-cr7t9\" (UID: \"bf23d9f0-60cd-49e2-b943-098589373c45\") " pod="openshift-console/downloads-68c9cbb497-cr7t9"
Jul 16 17:35:52.794022 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.793998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15ca0e4b-940b-48b3-8723-5bdf9de14ac3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75f94864-9z6dm\" (UID: \"15ca0e4b-940b-48b3-8723-5bdf9de14ac3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"
Jul 16 17:35:52.798400 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.798370    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-869b4d4848-dpsm9"]
Jul 16 17:35:52.800662 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.800642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.803922 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.803902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-59kcl\""
Jul 16 17:35:52.804024 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.803903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 17:35:52.804440 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.804412    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 17:35:52.804677 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.804662    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 17:35:52.811440 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.811421    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 17:35:52.813372 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.813350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26k58\" (UniqueName: \"kubernetes.io/projected/15ca0e4b-940b-48b3-8723-5bdf9de14ac3-kube-api-access-26k58\") pod \"managed-serviceaccount-addon-agent-75f94864-9z6dm\" (UID: \"15ca0e4b-940b-48b3-8723-5bdf9de14ac3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"
Jul 16 17:35:52.816464 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.816436    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869b4d4848-dpsm9"]
Jul 16 17:35:52.892335 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.892241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93efcccf-8ddb-461e-a1cd-a9a901480513-bound-sa-token\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.892335 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.892305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b2rd\" (UniqueName: \"kubernetes.io/projected/bf23d9f0-60cd-49e2-b943-098589373c45-kube-api-access-5b2rd\") pod \"downloads-68c9cbb497-cr7t9\" (UID: \"bf23d9f0-60cd-49e2-b943-098589373c45\") " pod="openshift-console/downloads-68c9cbb497-cr7t9"
Jul 16 17:35:52.892335 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.892337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93efcccf-8ddb-461e-a1cd-a9a901480513-registry-tls\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.892632 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.892357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hkj6\" (UniqueName: \"kubernetes.io/projected/93efcccf-8ddb-461e-a1cd-a9a901480513-kube-api-access-5hkj6\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.892632 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.892376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93efcccf-8ddb-461e-a1cd-a9a901480513-trusted-ca\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.892632 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.892423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93efcccf-8ddb-461e-a1cd-a9a901480513-ca-trust-extracted\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.892632 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.892446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93efcccf-8ddb-461e-a1cd-a9a901480513-registry-certificates\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.892632 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.892492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93efcccf-8ddb-461e-a1cd-a9a901480513-installation-pull-secrets\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.892632 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.892520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93efcccf-8ddb-461e-a1cd-a9a901480513-image-registry-private-configuration\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.906169 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.906130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b2rd\" (UniqueName: \"kubernetes.io/projected/bf23d9f0-60cd-49e2-b943-098589373c45-kube-api-access-5b2rd\") pod \"downloads-68c9cbb497-cr7t9\" (UID: \"bf23d9f0-60cd-49e2-b943-098589373c45\") " pod="openshift-console/downloads-68c9cbb497-cr7t9"
Jul 16 17:35:52.923027 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.922987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"
Jul 16 17:35:52.993356 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.993322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93efcccf-8ddb-461e-a1cd-a9a901480513-bound-sa-token\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.993522 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.993410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93efcccf-8ddb-461e-a1cd-a9a901480513-registry-tls\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.993522 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.993441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hkj6\" (UniqueName: \"kubernetes.io/projected/93efcccf-8ddb-461e-a1cd-a9a901480513-kube-api-access-5hkj6\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.993522 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.993471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93efcccf-8ddb-461e-a1cd-a9a901480513-trusted-ca\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.993719 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.993523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93efcccf-8ddb-461e-a1cd-a9a901480513-ca-trust-extracted\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.993719 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.993563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93efcccf-8ddb-461e-a1cd-a9a901480513-registry-certificates\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.993719 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.993606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93efcccf-8ddb-461e-a1cd-a9a901480513-installation-pull-secrets\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.993719 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.993633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93efcccf-8ddb-461e-a1cd-a9a901480513-image-registry-private-configuration\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.994140 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.994088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93efcccf-8ddb-461e-a1cd-a9a901480513-ca-trust-extracted\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.994498 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.994467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93efcccf-8ddb-461e-a1cd-a9a901480513-registry-certificates\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.994913 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.994889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93efcccf-8ddb-461e-a1cd-a9a901480513-trusted-ca\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.996036 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.996015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93efcccf-8ddb-461e-a1cd-a9a901480513-registry-tls\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.996251 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.996221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93efcccf-8ddb-461e-a1cd-a9a901480513-image-registry-private-configuration\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:52.996377 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:52.996356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93efcccf-8ddb-461e-a1cd-a9a901480513-installation-pull-secrets\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:53.002592 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.002566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hkj6\" (UniqueName: \"kubernetes.io/projected/93efcccf-8ddb-461e-a1cd-a9a901480513-kube-api-access-5hkj6\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:53.009112 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.009038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-cr7t9"
Jul 16 17:35:53.009272 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.009251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93efcccf-8ddb-461e-a1cd-a9a901480513-bound-sa-token\") pod \"image-registry-869b4d4848-dpsm9\" (UID: \"93efcccf-8ddb-461e-a1cd-a9a901480513\") " pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:53.064349 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.064212    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm"]
Jul 16 17:35:53.066909 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:35:53.066874    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15ca0e4b_940b_48b3_8723_5bdf9de14ac3.slice/crio-055fe11398b03213afe29fd5f88eaa86f0a1775eec729720f1ab38eeb5db9914 WatchSource:0}: Error finding container 055fe11398b03213afe29fd5f88eaa86f0a1775eec729720f1ab38eeb5db9914: Status 404 returned error can't find the container with id 055fe11398b03213afe29fd5f88eaa86f0a1775eec729720f1ab38eeb5db9914
Jul 16 17:35:53.109805 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.109775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:53.119436 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.119400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm" event={"ID":"15ca0e4b-940b-48b3-8723-5bdf9de14ac3","Type":"ContainerStarted","Data":"055fe11398b03213afe29fd5f88eaa86f0a1775eec729720f1ab38eeb5db9914"}
Jul 16 17:35:53.163418 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.163382    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-cr7t9"]
Jul 16 17:35:53.167596 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:35:53.167524    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf23d9f0_60cd_49e2_b943_098589373c45.slice/crio-42329f86adc7684efb95c858dab9a109c2d0f0690a722e9773f02ee9ad9ea9f2 WatchSource:0}: Error finding container 42329f86adc7684efb95c858dab9a109c2d0f0690a722e9773f02ee9ad9ea9f2: Status 404 returned error can't find the container with id 42329f86adc7684efb95c858dab9a109c2d0f0690a722e9773f02ee9ad9ea9f2
Jul 16 17:35:53.259204 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:35:53.259176    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93efcccf_8ddb_461e_a1cd_a9a901480513.slice/crio-d844b0418f6434176f451b530cc5973d659abc007d01bbd4b8a15bbe1046aba5 WatchSource:0}: Error finding container d844b0418f6434176f451b530cc5973d659abc007d01bbd4b8a15bbe1046aba5: Status 404 returned error can't find the container with id d844b0418f6434176f451b530cc5973d659abc007d01bbd4b8a15bbe1046aba5
Jul 16 17:35:53.259813 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.259790    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869b4d4848-dpsm9"]
Jul 16 17:35:53.397844 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.397806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:35:53.400132 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.400104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c928af6c-6ba7-40bb-8e7a-4a418754b963-metrics-certs\") pod \"network-metrics-daemon-gn4pn\" (UID: \"c928af6c-6ba7-40bb-8e7a-4a418754b963\") " pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:35:53.498374 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.498340    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2qkbl\""
Jul 16 17:35:53.504248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.504211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gn4pn"
Jul 16 17:35:53.657348 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.657307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gn4pn"]
Jul 16 17:35:53.661015 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:35:53.660984    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc928af6c_6ba7_40bb_8e7a_4a418754b963.slice/crio-915b0ddfdcddafbeb493b7724ebad8eb6f4695905da6246d5d85511d08ce1238 WatchSource:0}: Error finding container 915b0ddfdcddafbeb493b7724ebad8eb6f4695905da6246d5d85511d08ce1238: Status 404 returned error can't find the container with id 915b0ddfdcddafbeb493b7724ebad8eb6f4695905da6246d5d85511d08ce1238
Jul 16 17:35:53.801956 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.801873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:53.801956 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.801928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:53.802741 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.802686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5569cfbb-8a85-4f59-b995-32f0a18d7857-service-ca-bundle\") pod \"router-default-5cb66c8d8-nn55n\" (UID: \"5569cfbb-8a85-4f59-b995-32f0a18d7857\") " pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:53.805849 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:53.805821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54fdfddd-0dd4-4af2-90bc-ccaac4996484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v4b27\" (UID: \"54fdfddd-0dd4-4af2-90bc-ccaac4996484\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:54.060899 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.060803    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lx67b\""
Jul 16 17:35:54.064824 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.064795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dzqkw\""
Jul 16 17:35:54.068778 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.068208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:54.071986 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.071960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"
Jul 16 17:35:54.128148 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.128014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gn4pn" event={"ID":"c928af6c-6ba7-40bb-8e7a-4a418754b963","Type":"ContainerStarted","Data":"915b0ddfdcddafbeb493b7724ebad8eb6f4695905da6246d5d85511d08ce1238"}
Jul 16 17:35:54.131405 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.130423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869b4d4848-dpsm9" event={"ID":"93efcccf-8ddb-461e-a1cd-a9a901480513","Type":"ContainerStarted","Data":"961d9ea3887f3b99c48552f122db36f6ffae4eafbe9990745af374a9f50e0d7a"}
Jul 16 17:35:54.131405 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.130482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869b4d4848-dpsm9" event={"ID":"93efcccf-8ddb-461e-a1cd-a9a901480513","Type":"ContainerStarted","Data":"d844b0418f6434176f451b530cc5973d659abc007d01bbd4b8a15bbe1046aba5"}
Jul 16 17:35:54.131405 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.131341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:35:54.134843 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.134762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-cr7t9" event={"ID":"bf23d9f0-60cd-49e2-b943-098589373c45","Type":"ContainerStarted","Data":"42329f86adc7684efb95c858dab9a109c2d0f0690a722e9773f02ee9ad9ea9f2"}
Jul 16 17:35:54.171795 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.170876    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-869b4d4848-dpsm9" podStartSLOduration=2.17085274 podStartE2EDuration="2.17085274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:54.168843578 +0000 UTC m=+131.155379187" watchObservedRunningTime="2026-07-16 17:35:54.17085274 +0000 UTC m=+131.157388324"
Jul 16 17:35:54.289482 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.289430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cb66c8d8-nn55n"]
Jul 16 17:35:54.290634 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:35:54.290600    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5569cfbb_8a85_4f59_b995_32f0a18d7857.slice/crio-e7cb016c3f8658f316f72d34105d0ea2c5d4458c6c0986223d4354b34e3973e1 WatchSource:0}: Error finding container e7cb016c3f8658f316f72d34105d0ea2c5d4458c6c0986223d4354b34e3973e1: Status 404 returned error can't find the container with id e7cb016c3f8658f316f72d34105d0ea2c5d4458c6c0986223d4354b34e3973e1
Jul 16 17:35:54.306331 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:54.306116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27"]
Jul 16 17:35:54.311767 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:35:54.311700    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fdfddd_0dd4_4af2_90bc_ccaac4996484.slice/crio-256cc9077140633cffb8766544f32cce27ce0601e1e7009a002bdcc86495270f WatchSource:0}: Error finding container 256cc9077140633cffb8766544f32cce27ce0601e1e7009a002bdcc86495270f: Status 404 returned error can't find the container with id 256cc9077140633cffb8766544f32cce27ce0601e1e7009a002bdcc86495270f
Jul 16 17:35:55.142062 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:55.142013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gn4pn" event={"ID":"c928af6c-6ba7-40bb-8e7a-4a418754b963","Type":"ContainerStarted","Data":"7a20613d40ae8556de07ad07915e1e34d14e6908b60c704d9409ab14ab7874dc"}
Jul 16 17:35:55.142502 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:55.142086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gn4pn" event={"ID":"c928af6c-6ba7-40bb-8e7a-4a418754b963","Type":"ContainerStarted","Data":"67692a5d5a65ff9140e9e6b82aa70e927e9da2c3c4583a12d3ec7d7f990fa8c2"}
Jul 16 17:35:55.144314 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:55.144212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27" event={"ID":"54fdfddd-0dd4-4af2-90bc-ccaac4996484","Type":"ContainerStarted","Data":"256cc9077140633cffb8766544f32cce27ce0601e1e7009a002bdcc86495270f"}
Jul 16 17:35:55.146358 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:55.146275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cb66c8d8-nn55n" event={"ID":"5569cfbb-8a85-4f59-b995-32f0a18d7857","Type":"ContainerStarted","Data":"74792efaf521cf3e4bc705d1ee1a85d5188774a754d0f45f39bce4e5b5cf051d"}
Jul 16 17:35:55.146358 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:55.146324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cb66c8d8-nn55n" event={"ID":"5569cfbb-8a85-4f59-b995-32f0a18d7857","Type":"ContainerStarted","Data":"e7cb016c3f8658f316f72d34105d0ea2c5d4458c6c0986223d4354b34e3973e1"}
Jul 16 17:35:55.164438 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:55.164245    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gn4pn" podStartSLOduration=130.929993374 podStartE2EDuration="2m12.164223348s" totalImagesPullingTime="1.234229974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:55.163793612 +0000 UTC m=+132.150329219" watchObservedRunningTime="2026-07-16 17:35:55.164223348 +0000 UTC m=+132.150758956"
Jul 16 17:35:55.187894 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:55.187818    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5cb66c8d8-nn55n" podStartSLOduration=34.187794642 podStartE2EDuration="34.187794642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:55.186605892 +0000 UTC m=+132.173141499" watchObservedRunningTime="2026-07-16 17:35:55.187794642 +0000 UTC m=+132.174330251"
Jul 16 17:35:56.069194 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:56.069157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:56.071898 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:56.071872    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:56.150902 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:56.150819    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:56.151984 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:56.151959    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5cb66c8d8-nn55n"
Jul 16 17:35:57.158331 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:57.157899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27" event={"ID":"54fdfddd-0dd4-4af2-90bc-ccaac4996484","Type":"ContainerStarted","Data":"30b3fc81bdad81e250bebeef82dd51d7df3ab82dfc6267fae2563a6446b847a2"}
Jul 16 17:35:57.159951 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:57.159904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm" event={"ID":"15ca0e4b-940b-48b3-8723-5bdf9de14ac3","Type":"ContainerStarted","Data":"1894f4f1f4f8900560d3c08858fb9598a7d727f0a8667eb619790cdb8c8388d8"}
Jul 16 17:35:57.178242 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:57.178177    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27" podStartSLOduration=33.461494281 podStartE2EDuration="36.178155387s" totalImagesPullingTime="2.716661106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:57.175777366 +0000 UTC m=+134.162312974" watchObservedRunningTime="2026-07-16 17:35:57.178155387 +0000 UTC m=+134.164690997"
Jul 16 17:35:57.195905 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:57.195850    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f94864-9z6dm" podStartSLOduration=1.229526726 podStartE2EDuration="5.195831397s" totalImagesPullingTime="3.966304671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:57.195815793 +0000 UTC m=+134.182351400" watchObservedRunningTime="2026-07-16 17:35:57.195831397 +0000 UTC m=+134.182367002"
Jul 16 17:35:57.738574 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:57.738521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:57.741294 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:57.741261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80ceb78a-1472-4ff4-889f-dc7a16520c1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qwdm\" (UID: \"80ceb78a-1472-4ff4-889f-dc7a16520c1c\") " pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:57.939610 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:57.939579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nsb9l\""
Jul 16 17:35:57.946711 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:57.946685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qwdm"
Jul 16 17:35:58.094267 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:58.094230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qwdm"]
Jul 16 17:35:58.098584 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:35:58.098537    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ceb78a_1472_4ff4_889f_dc7a16520c1c.slice/crio-4a4cf2af39e527d145ee03529f2d95b202edcaa1ddfe0838381fbb0729c5dbbc WatchSource:0}: Error finding container 4a4cf2af39e527d145ee03529f2d95b202edcaa1ddfe0838381fbb0729c5dbbc: Status 404 returned error can't find the container with id 4a4cf2af39e527d145ee03529f2d95b202edcaa1ddfe0838381fbb0729c5dbbc
Jul 16 17:35:58.164917 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:58.164862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qwdm" event={"ID":"80ceb78a-1472-4ff4-889f-dc7a16520c1c","Type":"ContainerStarted","Data":"4a4cf2af39e527d145ee03529f2d95b202edcaa1ddfe0838381fbb0729c5dbbc"}
Jul 16 17:35:59.678272 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.678241    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ljk48"]
Jul 16 17:35:59.680915 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.680892    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.684040 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.684000    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 17:35:59.684506 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.684443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 17:35:59.684506 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.684482    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hdbl6\""
Jul 16 17:35:59.685841 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.685820    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 17:35:59.697115 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.697082    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ljk48"]
Jul 16 17:35:59.754855 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.754781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vpk\" (UniqueName: \"kubernetes.io/projected/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-kube-api-access-v6vpk\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.755011 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.754941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.755011 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.754993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.755164 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.755036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.856517 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.856457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.856810 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.856522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.856810 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.856582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.856810 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.856633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vpk\" (UniqueName: \"kubernetes.io/projected/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-kube-api-access-v6vpk\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.857408 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:59.857097    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 16 17:35:59.857408 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:35:59.857175    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-prometheus-operator-tls podName:5b39b98b-0c71-4664-ac40-b0a6b4ce520e nodeName:}" failed. No retries permitted until 2026-07-16 17:36:00.357154009 +0000 UTC m=+137.343689592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-prometheus-operator-tls") pod "prometheus-operator-64785b6675-ljk48" (UID: "5b39b98b-0c71-4664-ac40-b0a6b4ce520e") : secret "prometheus-operator-tls" not found
Jul 16 17:35:59.857408 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.857317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.859996 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.859968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:35:59.867464 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:35:59.867401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vpk\" (UniqueName: \"kubernetes.io/projected/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-kube-api-access-v6vpk\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:36:00.361367 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:00.361318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:36:00.364441 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:00.364408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b39b98b-0c71-4664-ac40-b0a6b4ce520e-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ljk48\" (UID: \"5b39b98b-0c71-4664-ac40-b0a6b4ce520e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:36:00.593517 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:00.593477    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48"
Jul 16 17:36:00.756243 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:00.756201    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ljk48"]
Jul 16 17:36:03.047530 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:36:03.047489    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b39b98b_0c71_4664_ac40_b0a6b4ce520e.slice/crio-58f4e841695d68ffe1e9ca8b650abe84ca8b85972ef6fd8967534268195ae633 WatchSource:0}: Error finding container 58f4e841695d68ffe1e9ca8b650abe84ca8b85972ef6fd8967534268195ae633: Status 404 returned error can't find the container with id 58f4e841695d68ffe1e9ca8b650abe84ca8b85972ef6fd8967534268195ae633
Jul 16 17:36:03.189189 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:03.189139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48" event={"ID":"5b39b98b-0c71-4664-ac40-b0a6b4ce520e","Type":"ContainerStarted","Data":"58f4e841695d68ffe1e9ca8b650abe84ca8b85972ef6fd8967534268195ae633"}
Jul 16 17:36:11.222112 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:11.222069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qwdm" event={"ID":"80ceb78a-1472-4ff4-889f-dc7a16520c1c","Type":"ContainerStarted","Data":"32280e650c16d726a11a1d6fcd4e81abefd9369a33b9926d7ee647a4efbf81dc"}
Jul 16 17:36:11.224460 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:11.224217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-cr7t9" event={"ID":"bf23d9f0-60cd-49e2-b943-098589373c45","Type":"ContainerStarted","Data":"14d23a703a2ff2b76be02252d776bca7cd1d70f43d737130a948f88d44917e0b"}
Jul 16 17:36:11.224739 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:11.224714    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-cr7t9"
Jul 16 17:36:11.236406 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:11.236333    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-cr7t9"
Jul 16 17:36:11.247232 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:11.247168    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-cr7t9" podStartSLOduration=1.538643261 podStartE2EDuration="19.247143683s" totalImagesPullingTime="17.708500422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:11.246498195 +0000 UTC m=+148.233033806" watchObservedRunningTime="2026-07-16 17:36:11.247143683 +0000 UTC m=+148.233679292"
Jul 16 17:36:13.115797 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:13.115748    2563 patch_prober.go:28] interesting pod/image-registry-869b4d4848-dpsm9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 16 17:36:13.116308 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:13.115828    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-869b4d4848-dpsm9" podUID="93efcccf-8ddb-461e-a1cd-a9a901480513" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:36:13.236387 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:13.236339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48" event={"ID":"5b39b98b-0c71-4664-ac40-b0a6b4ce520e","Type":"ContainerStarted","Data":"60bc7631cff17a941bc0b9dd28142f099f029f50ad160bb6060e1951ee62585a"}
Jul 16 17:36:13.236640 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:13.236465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48" event={"ID":"5b39b98b-0c71-4664-ac40-b0a6b4ce520e","Type":"ContainerStarted","Data":"642c6f2f3a867ba782950b8159c9df13f1139dda1c6551f73227ab0807ee5b71"}
Jul 16 17:36:13.258383 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:13.258314    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-ljk48" podStartSLOduration=5.192423382 podStartE2EDuration="14.258290003s" totalImagesPullingTime="9.065866621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:13.25597646 +0000 UTC m=+150.242512067" watchObservedRunningTime="2026-07-16 17:36:13.258290003 +0000 UTC m=+150.244825614"
Jul 16 17:36:15.246880 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:15.246795    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:36:15.246880 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:15.246844    2563 generic.go:356] "Generic (PLEG): container finished" podID="54fdfddd-0dd4-4af2-90bc-ccaac4996484" containerID="30b3fc81bdad81e250bebeef82dd51d7df3ab82dfc6267fae2563a6446b847a2" exitCode=2
Jul 16 17:36:15.247366 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:15.246892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27" event={"ID":"54fdfddd-0dd4-4af2-90bc-ccaac4996484","Type":"ContainerDied","Data":"30b3fc81bdad81e250bebeef82dd51d7df3ab82dfc6267fae2563a6446b847a2"}
Jul 16 17:36:15.247366 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:15.247250    2563 scope.go:117] "RemoveContainer" containerID="30b3fc81bdad81e250bebeef82dd51d7df3ab82dfc6267fae2563a6446b847a2"
Jul 16 17:36:16.162363 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:16.162320    2563 patch_prober.go:28] interesting pod/image-registry-869b4d4848-dpsm9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 16 17:36:16.162527 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:16.162396    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-869b4d4848-dpsm9" podUID="93efcccf-8ddb-461e-a1cd-a9a901480513" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:36:16.253929 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:16.253902    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:36:16.254380 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:16.254037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v4b27" event={"ID":"54fdfddd-0dd4-4af2-90bc-ccaac4996484","Type":"ContainerStarted","Data":"e56d5488352f626ef91997b0048f7926b71782a58d918895fe5bd68e59fbccc7"}
Jul 16 17:36:16.256777 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:16.256750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qwdm" event={"ID":"80ceb78a-1472-4ff4-889f-dc7a16520c1c","Type":"ContainerStarted","Data":"85367d192cb967c2f667d9e95e9c45b0d5652e37dba20084104c4ff125487ac5"}
Jul 16 17:36:19.656391 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.656084    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-69lnj"]
Jul 16 17:36:19.827039 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.826999    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:19.831028 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.830988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zznr4\""
Jul 16 17:36:19.832664 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.832634    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 17:36:19.832664 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.832654    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 17:36:19.832924 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.832907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 17:36:19.948269 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.948113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/657cd3e3-3c63-4709-a8ed-885d35227bfa-metrics-client-ca\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:19.948269 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.948184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/657cd3e3-3c63-4709-a8ed-885d35227bfa-sys\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:19.948269 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.948215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:19.948269 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.948243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:19.948649 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.948283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-tls\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:19.948649 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.948333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/657cd3e3-3c63-4709-a8ed-885d35227bfa-root\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:19.948649 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.948355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd4bg\" (UniqueName: \"kubernetes.io/projected/657cd3e3-3c63-4709-a8ed-885d35227bfa-kube-api-access-kd4bg\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:19.948649 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.948392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-textfile\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:19.948649 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:19.948424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-wtmp\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.049117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd4bg\" (UniqueName: \"kubernetes.io/projected/657cd3e3-3c63-4709-a8ed-885d35227bfa-kube-api-access-kd4bg\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.049189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-textfile\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.049228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-wtmp\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.049272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/657cd3e3-3c63-4709-a8ed-885d35227bfa-metrics-client-ca\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.049316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/657cd3e3-3c63-4709-a8ed-885d35227bfa-sys\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.049348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.049376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.049414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-tls\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.049475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/657cd3e3-3c63-4709-a8ed-885d35227bfa-root\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.049587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/657cd3e3-3c63-4709-a8ed-885d35227bfa-root\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.050229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-textfile\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.050672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.050370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-wtmp\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.051415 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.051219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.051415 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.051286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/657cd3e3-3c63-4709-a8ed-885d35227bfa-sys\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.052669 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.052619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/657cd3e3-3c63-4709-a8ed-885d35227bfa-metrics-client-ca\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.054583 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.054125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-tls\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.054583 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.054147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/657cd3e3-3c63-4709-a8ed-885d35227bfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.062684 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.062613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd4bg\" (UniqueName: \"kubernetes.io/projected/657cd3e3-3c63-4709-a8ed-885d35227bfa-kube-api-access-kd4bg\") pod \"node-exporter-69lnj\" (UID: \"657cd3e3-3c63-4709-a8ed-885d35227bfa\") " pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.141838 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:20.141795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-69lnj"
Jul 16 17:36:20.464243 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:36:20.464206    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657cd3e3_3c63_4709_a8ed_885d35227bfa.slice/crio-5e7caf7c0e748bf6db7ca918e62e51af87acd3d2ebc7dfac60352e0b34c99084 WatchSource:0}: Error finding container 5e7caf7c0e748bf6db7ca918e62e51af87acd3d2ebc7dfac60352e0b34c99084: Status 404 returned error can't find the container with id 5e7caf7c0e748bf6db7ca918e62e51af87acd3d2ebc7dfac60352e0b34c99084
Jul 16 17:36:20.896885 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:36:20.896844    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nqwmf" podUID="b1836785-46a4-4231-a977-eee3a2d8b563"
Jul 16 17:36:21.280946 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:21.280902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qwdm" event={"ID":"80ceb78a-1472-4ff4-889f-dc7a16520c1c","Type":"ContainerStarted","Data":"d4aa9b59edf487ffc0dbdf5b66354badc09f2fb891d7cf5bc7bac174be370129"}
Jul 16 17:36:21.282207 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:21.282187    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:36:21.282314 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:21.282202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-69lnj" event={"ID":"657cd3e3-3c63-4709-a8ed-885d35227bfa","Type":"ContainerStarted","Data":"5e7caf7c0e748bf6db7ca918e62e51af87acd3d2ebc7dfac60352e0b34c99084"}
Jul 16 17:36:21.304410 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:21.304350    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7qwdm" podStartSLOduration=33.757963175 podStartE2EDuration="56.304325515s" totalImagesPullingTime="22.54636234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:35:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:21.301910422 +0000 UTC m=+158.288446027" watchObservedRunningTime="2026-07-16 17:36:21.304325515 +0000 UTC m=+158.290861123"
Jul 16 17:36:22.638069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.637975    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"]
Jul 16 17:36:22.657807 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.657766    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"]
Jul 16 17:36:22.657989 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.657882    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.660901 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.660867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 16 17:36:22.661043 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.660933    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 16 17:36:22.661197 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.661181    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 16 17:36:22.661349 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.661326    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5mcwr\""
Jul 16 17:36:22.661461 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.661373    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 16 17:36:22.662160 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.662142    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 16 17:36:22.662160 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.662150    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6cmponof1148u\""
Jul 16 17:36:22.772703 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.772652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.772703 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.772711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-grpc-tls\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.772931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.772744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-tls\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.772931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.772762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.772931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.772780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.772931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.772866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8d4f\" (UniqueName: \"kubernetes.io/projected/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-kube-api-access-l8d4f\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.772931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.772906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.772931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.772930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-metrics-client-ca\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.874211 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.874171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8d4f\" (UniqueName: \"kubernetes.io/projected/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-kube-api-access-l8d4f\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.874387 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.874223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.874387 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.874248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-metrics-client-ca\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.874387 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.874311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.874387 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.874350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-grpc-tls\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.874387 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.874378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-tls\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.874679 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.874406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.874679 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.874429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.875156 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.875123    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-metrics-client-ca\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.877139 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.877094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.877275 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.877139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.877339 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.877310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-tls\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.877339 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.877327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.877521 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.877490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.877645 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.877621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-secret-grpc-tls\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.888046 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.888013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8d4f\" (UniqueName: \"kubernetes.io/projected/b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa-kube-api-access-l8d4f\") pod \"thanos-querier-6ccd95c578-xn2c9\" (UID: \"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa\") " pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:22.966897 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:22.966804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:23.114903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:23.114861    2563 patch_prober.go:28] interesting pod/image-registry-869b4d4848-dpsm9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 16 17:36:23.115069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:23.114937    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-869b4d4848-dpsm9" podUID="93efcccf-8ddb-461e-a1cd-a9a901480513" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:36:23.151558 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:23.151522    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"]
Jul 16 17:36:23.153226 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:36:23.153202    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f2e90e_67ed_4fe1_bbaf_9b15e2ba3bfa.slice/crio-2b00179f395a45fe4d578bef7164d2daaf1faa47dda8a5a3f240f49f9eca18cd WatchSource:0}: Error finding container 2b00179f395a45fe4d578bef7164d2daaf1faa47dda8a5a3f240f49f9eca18cd: Status 404 returned error can't find the container with id 2b00179f395a45fe4d578bef7164d2daaf1faa47dda8a5a3f240f49f9eca18cd
Jul 16 17:36:23.291960 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:23.291922    2563 generic.go:356] "Generic (PLEG): container finished" podID="657cd3e3-3c63-4709-a8ed-885d35227bfa" containerID="ce5c85afb454fe70bd115eecf3df0506048ae99a004aa1ec384b56d5925c6927" exitCode=0
Jul 16 17:36:23.292177 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:23.292007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-69lnj" event={"ID":"657cd3e3-3c63-4709-a8ed-885d35227bfa","Type":"ContainerDied","Data":"ce5c85afb454fe70bd115eecf3df0506048ae99a004aa1ec384b56d5925c6927"}
Jul 16 17:36:23.293304 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:23.293272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9" event={"ID":"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa","Type":"ContainerStarted","Data":"2b00179f395a45fe4d578bef7164d2daaf1faa47dda8a5a3f240f49f9eca18cd"}
Jul 16 17:36:24.300158 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.300124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-69lnj" event={"ID":"657cd3e3-3c63-4709-a8ed-885d35227bfa","Type":"ContainerStarted","Data":"b6f4e4a81c5982757076a6f7dade96e61b8baf7c1105040abf1d337db33a0f4b"}
Jul 16 17:36:24.300649 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.300185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-69lnj" event={"ID":"657cd3e3-3c63-4709-a8ed-885d35227bfa","Type":"ContainerStarted","Data":"3b73dd102bcf6c5a8b56d4c73022e12d5654fe1f2723e796d5be8e9647509088"}
Jul 16 17:36:24.876204 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.876146    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-69lnj" podStartSLOduration=3.949271892 podStartE2EDuration="5.876124813s" totalImagesPullingTime="1.854563672s" totalInitContainerRuntime="72.289249ms" isStatefulPod=true podCreationTimestamp="2026-07-16 17:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:24.331330123 +0000 UTC m=+161.317865738" watchObservedRunningTime="2026-07-16 17:36:24.876124813 +0000 UTC m=+161.862660418"
Jul 16 17:36:24.876394 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.876331    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"]
Jul 16 17:36:24.900406 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.900368    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"]
Jul 16 17:36:24.900651 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.900508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:24.907919 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.907893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 16 17:36:24.908050 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.907893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 16 17:36:24.908050 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.907893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 16 17:36:24.908050 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.907893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g4hpj\""
Jul 16 17:36:24.908050 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.908042    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 16 17:36:24.909320 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.909298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 16 17:36:24.913245 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.913222    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 16 17:36:24.992387 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.992352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-telemeter-client-tls\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:24.992587 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.992392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:24.992587 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.992418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-federate-client-tls\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:24.992587 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.992492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrst\" (UniqueName: \"kubernetes.io/projected/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-kube-api-access-cbrst\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:24.992587 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.992533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-serving-certs-ca-bundle\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:24.992730 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.992588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-secret-telemeter-client\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:24.992730 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.992620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:24.992730 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:24.992636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-metrics-client-ca\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.093211 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.093172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.093211 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.093217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-federate-client-tls\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.093450 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.093252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrst\" (UniqueName: \"kubernetes.io/projected/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-kube-api-access-cbrst\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.093450 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.093269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-serving-certs-ca-bundle\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.093450 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.093288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-secret-telemeter-client\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.093450 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.093311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.093450 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.093328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-metrics-client-ca\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.093450 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.093398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-telemeter-client-tls\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.094411 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.094144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-serving-certs-ca-bundle\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.094411 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.094219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-metrics-client-ca\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.094411 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.094268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.096157 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.096132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-federate-client-tls\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.096381 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.096359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.096456 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.096442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-telemeter-client-tls\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.096526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.096497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-secret-telemeter-client\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.105589 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.105536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrst\" (UniqueName: \"kubernetes.io/projected/0ea7c990-25ec-4785-bf6e-4f5e5f2cc847-kube-api-access-cbrst\") pod \"telemeter-client-6cf68d7f88-6vtrb\" (UID: \"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847\") " pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.210076 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.209993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"
Jul 16 17:36:25.394335 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.394243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb"]
Jul 16 17:36:25.401290 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:36:25.401260    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea7c990_25ec_4785_bf6e_4f5e5f2cc847.slice/crio-107043f361dbf8786d09cab57ec08b3e67d154cbf107196aa13947bdf2e5e800 WatchSource:0}: Error finding container 107043f361dbf8786d09cab57ec08b3e67d154cbf107196aa13947bdf2e5e800: Status 404 returned error can't find the container with id 107043f361dbf8786d09cab57ec08b3e67d154cbf107196aa13947bdf2e5e800
Jul 16 17:36:25.900743 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.900699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:36:25.903274 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:25.903233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1836785-46a4-4231-a977-eee3a2d8b563-metrics-tls\") pod \"dns-default-nqwmf\" (UID: \"b1836785-46a4-4231-a977-eee3a2d8b563\") " pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:36:26.087031 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:26.086996    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-24sj8\""
Jul 16 17:36:26.094102 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:26.094067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:36:26.158348 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:26.158261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-869b4d4848-dpsm9"
Jul 16 17:36:26.310871 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:26.310815    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nqwmf"]
Jul 16 17:36:26.315125 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:26.315085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9" event={"ID":"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa","Type":"ContainerStarted","Data":"3a87f0f70af6a23e82649c7e287a26f3f51ffb0236cdf371a710772152bf0f27"}
Jul 16 17:36:26.315237 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:26.315187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9" event={"ID":"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa","Type":"ContainerStarted","Data":"26277f1f3ddba847a3f00c5be8067df01bde8f13d7566226fa8888f09de32e2c"}
Jul 16 17:36:26.315237 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:26.315231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9" event={"ID":"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa","Type":"ContainerStarted","Data":"4147b90c1765f3df388ec3b540ffbd87bdad866282e9eb9e0eb055fd298e43f4"}
Jul 16 17:36:26.317229 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:26.316709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb" event={"ID":"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847","Type":"ContainerStarted","Data":"107043f361dbf8786d09cab57ec08b3e67d154cbf107196aa13947bdf2e5e800"}
Jul 16 17:36:26.317703 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:36:26.317676    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1836785_46a4_4231_a977_eee3a2d8b563.slice/crio-fbbe102bb3d99e49a1d22d870d2161d99333f3414855177cff79b4e76e39756f WatchSource:0}: Error finding container fbbe102bb3d99e49a1d22d870d2161d99333f3414855177cff79b4e76e39756f: Status 404 returned error can't find the container with id fbbe102bb3d99e49a1d22d870d2161d99333f3414855177cff79b4e76e39756f
Jul 16 17:36:27.326380 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:27.326305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nqwmf" event={"ID":"b1836785-46a4-4231-a977-eee3a2d8b563","Type":"ContainerStarted","Data":"fbbe102bb3d99e49a1d22d870d2161d99333f3414855177cff79b4e76e39756f"}
Jul 16 17:36:29.337224 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:29.337181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nqwmf" event={"ID":"b1836785-46a4-4231-a977-eee3a2d8b563","Type":"ContainerStarted","Data":"b3fe3f79dfe8a7615557ca4e7ff725f6d7995cede76ef705a0b3f64356289e03"}
Jul 16 17:36:29.338825 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:29.338799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb" event={"ID":"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847","Type":"ContainerStarted","Data":"8bdafe95008213740f4cef2ee2f5b3be2f9c4d0a681d7d24952c308c83033cf7"}
Jul 16 17:36:30.344606 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:30.344524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nqwmf" event={"ID":"b1836785-46a4-4231-a977-eee3a2d8b563","Type":"ContainerStarted","Data":"5a004170edab247cf029dc3800fc9c2659fa8e277f358486e87d3a833d5dcf32"}
Jul 16 17:36:30.345156 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:30.344711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:36:30.347293 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:30.347266    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9" event={"ID":"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa","Type":"ContainerStarted","Data":"b61b6306caa99dc2acc2d2aab325efc47c436ca3a87c12cbd6f219dfa6149371"}
Jul 16 17:36:30.347408 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:30.347338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9" event={"ID":"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa","Type":"ContainerStarted","Data":"e00a90c879956e633fba155fdc33ab35217c2cf94c46df2e1403f2fabc834bde"}
Jul 16 17:36:30.347408 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:30.347366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9" event={"ID":"b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa","Type":"ContainerStarted","Data":"7a16e98ea20a0294c79f5990256db1a59e29470cc76bea3a4101a223bab0f39a"}
Jul 16 17:36:30.347562 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:30.347538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:30.353069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:30.353044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9"
Jul 16 17:36:30.364460 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:30.364409    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nqwmf" podStartSLOduration=130.741687223 podStartE2EDuration="2m13.364392502s" totalImagesPullingTime="2.622705279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:30.364148906 +0000 UTC m=+167.350684513" watchObservedRunningTime="2026-07-16 17:36:30.364392502 +0000 UTC m=+167.350928108"
Jul 16 17:36:30.388348 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:30.388297    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6ccd95c578-xn2c9" podStartSLOduration=2.687698705 podStartE2EDuration="8.3882788s" totalImagesPullingTime="5.700580095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:30.386908822 +0000 UTC m=+167.373444440" watchObservedRunningTime="2026-07-16 17:36:30.3882788 +0000 UTC m=+167.374814406"
Jul 16 17:36:31.357755 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:31.357717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb" event={"ID":"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847","Type":"ContainerStarted","Data":"07d8f995ad192b4be0518da0b707dbea8b9b4a73714af219fab7f293b15ea9a9"}
Jul 16 17:36:31.358257 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:31.357803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb" event={"ID":"0ea7c990-25ec-4785-bf6e-4f5e5f2cc847","Type":"ContainerStarted","Data":"715b287e39ae879d2a2e91b814e3bf964e00aadd6152b35a50a49c9ee50f5408"}
Jul 16 17:36:31.386271 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:31.386203    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cf68d7f88-6vtrb" podStartSLOduration=2.307008596 podStartE2EDuration="7.386183822s" totalImagesPullingTime="5.079175226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:31.385142452 +0000 UTC m=+168.371678057" watchObservedRunningTime="2026-07-16 17:36:31.386183822 +0000 UTC m=+168.372719427"
Jul 16 17:36:40.360793 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:40.360764    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nqwmf"
Jul 16 17:36:41.285707 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:41.285679    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nqwmf_b1836785-46a4-4231-a977-eee3a2d8b563/dns/0.log"
Jul 16 17:36:41.293369 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:41.293339    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nqwmf_b1836785-46a4-4231-a977-eee3a2d8b563/kube-rbac-proxy/0.log"
Jul 16 17:36:41.765289 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:36:41.765263    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mxzqv_fa1f394b-ac23-4640-86ed-75f04466fa1e/dns-node-resolver/0.log"
Jul 16 17:38:43.489378 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:38:43.489350    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:38:43.489875 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:38:43.489356    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:38:43.492763 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:38:43.492731    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:38:43.492763 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:38:43.492755    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:38:43.497449 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:38:43.497425    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:38:43.497685 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:38:43.497669    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:38:43.504115 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:38:43.504091    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 17:39:42.128945 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.128908    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"]
Jul 16 17:39:42.131444 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.131426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.134582 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.134545    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 17:39:42.135772 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.135749    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 17:39:42.135883 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.135756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-576lr\""
Jul 16 17:39:42.141515 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.141483    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"]
Jul 16 17:39:42.180653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.180619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26j7j\" (UniqueName: \"kubernetes.io/projected/1cdee90c-09c5-4f7c-891d-d313d806c5e3-kube-api-access-26j7j\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.180849 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.180666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.180849 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.180767    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.281917 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.281880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26j7j\" (UniqueName: \"kubernetes.io/projected/1cdee90c-09c5-4f7c-891d-d313d806c5e3-kube-api-access-26j7j\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.282018 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.281929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.282018 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.281995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.282423 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.282401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.282495 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.282424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.291825 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.291797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26j7j\" (UniqueName: \"kubernetes.io/projected/1cdee90c-09c5-4f7c-891d-d313d806c5e3-kube-api-access-26j7j\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.440599 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.440463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:39:42.582693 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.582665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"]
Jul 16 17:39:42.585320 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:39:42.585285    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdee90c_09c5_4f7c_891d_d313d806c5e3.slice/crio-35040b06a58c633660b2c0f2acc73b2cb96850bd680ce9081eb57236ae6260e2 WatchSource:0}: Error finding container 35040b06a58c633660b2c0f2acc73b2cb96850bd680ce9081eb57236ae6260e2: Status 404 returned error can't find the container with id 35040b06a58c633660b2c0f2acc73b2cb96850bd680ce9081eb57236ae6260e2
Jul 16 17:39:42.587697 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:42.587679    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:39:43.097688 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:43.097648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs" event={"ID":"1cdee90c-09c5-4f7c-891d-d313d806c5e3","Type":"ContainerStarted","Data":"35040b06a58c633660b2c0f2acc73b2cb96850bd680ce9081eb57236ae6260e2"}
Jul 16 17:39:51.132072 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:51.132035    2563 generic.go:356] "Generic (PLEG): container finished" podID="1cdee90c-09c5-4f7c-891d-d313d806c5e3" containerID="058dcff17b1ff92ffabdcb33a44f204577311dbd6add4c6e9150bb1abaab9ecd" exitCode=0
Jul 16 17:39:51.132487 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:51.132123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs" event={"ID":"1cdee90c-09c5-4f7c-891d-d313d806c5e3","Type":"ContainerDied","Data":"058dcff17b1ff92ffabdcb33a44f204577311dbd6add4c6e9150bb1abaab9ecd"}
Jul 16 17:39:54.145121 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:54.145085    2563 generic.go:356] "Generic (PLEG): container finished" podID="1cdee90c-09c5-4f7c-891d-d313d806c5e3" containerID="5601fa0fc87a30f0e8031388a5b44c8c8ba0d4266ed35ae881972e2fac44cbb8" exitCode=0
Jul 16 17:39:54.145532 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:39:54.145184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs" event={"ID":"1cdee90c-09c5-4f7c-891d-d313d806c5e3","Type":"ContainerDied","Data":"5601fa0fc87a30f0e8031388a5b44c8c8ba0d4266ed35ae881972e2fac44cbb8"}
Jul 16 17:40:01.179249 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:01.179213    2563 generic.go:356] "Generic (PLEG): container finished" podID="1cdee90c-09c5-4f7c-891d-d313d806c5e3" containerID="8b5f0918141fa58de2bb3340f9c33dc7cacc1f961ba1a8e74346d8af0645a2a0" exitCode=0
Jul 16 17:40:01.179663 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:01.179265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs" event={"ID":"1cdee90c-09c5-4f7c-891d-d313d806c5e3","Type":"ContainerDied","Data":"8b5f0918141fa58de2bb3340f9c33dc7cacc1f961ba1a8e74346d8af0645a2a0"}
Jul 16 17:40:02.311371 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:02.311345    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:40:02.358423 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:02.358385    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-bundle\") pod \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") "
Jul 16 17:40:02.358639 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:02.358438    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26j7j\" (UniqueName: \"kubernetes.io/projected/1cdee90c-09c5-4f7c-891d-d313d806c5e3-kube-api-access-26j7j\") pod \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") "
Jul 16 17:40:02.358639 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:02.358477    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-util\") pod \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\" (UID: \"1cdee90c-09c5-4f7c-891d-d313d806c5e3\") "
Jul 16 17:40:02.359095 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:02.359068    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-bundle" (OuterVolumeSpecName: "bundle") pod "1cdee90c-09c5-4f7c-891d-d313d806c5e3" (UID: "1cdee90c-09c5-4f7c-891d-d313d806c5e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:40:02.360712 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:02.360688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdee90c-09c5-4f7c-891d-d313d806c5e3-kube-api-access-26j7j" (OuterVolumeSpecName: "kube-api-access-26j7j") pod "1cdee90c-09c5-4f7c-891d-d313d806c5e3" (UID: "1cdee90c-09c5-4f7c-891d-d313d806c5e3"). InnerVolumeSpecName "kube-api-access-26j7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:40:02.363294 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:02.363268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-util" (OuterVolumeSpecName: "util") pod "1cdee90c-09c5-4f7c-891d-d313d806c5e3" (UID: "1cdee90c-09c5-4f7c-891d-d313d806c5e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:40:02.459671 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:02.459538    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-util\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:40:02.459671 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:02.459630    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cdee90c-09c5-4f7c-891d-d313d806c5e3-bundle\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:40:02.459671 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:02.459642    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26j7j\" (UniqueName: \"kubernetes.io/projected/1cdee90c-09c5-4f7c-891d-d313d806c5e3-kube-api-access-26j7j\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:40:03.188458 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:03.188424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs" event={"ID":"1cdee90c-09c5-4f7c-891d-d313d806c5e3","Type":"ContainerDied","Data":"35040b06a58c633660b2c0f2acc73b2cb96850bd680ce9081eb57236ae6260e2"}
Jul 16 17:40:03.188690 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:03.188464    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f9czs"
Jul 16 17:40:03.188690 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:03.188469    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35040b06a58c633660b2c0f2acc73b2cb96850bd680ce9081eb57236ae6260e2"
Jul 16 17:40:14.964948 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.964912    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-7hzkd"]
Jul 16 17:40:14.965391 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.965196    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdee90c-09c5-4f7c-891d-d313d806c5e3" containerName="pull"
Jul 16 17:40:14.965391 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.965207    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdee90c-09c5-4f7c-891d-d313d806c5e3" containerName="pull"
Jul 16 17:40:14.965391 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.965227    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdee90c-09c5-4f7c-891d-d313d806c5e3" containerName="util"
Jul 16 17:40:14.965391 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.965233    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdee90c-09c5-4f7c-891d-d313d806c5e3" containerName="util"
Jul 16 17:40:14.965391 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.965239    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdee90c-09c5-4f7c-891d-d313d806c5e3" containerName="extract"
Jul 16 17:40:14.965391 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.965245    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdee90c-09c5-4f7c-891d-d313d806c5e3" containerName="extract"
Jul 16 17:40:14.965391 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.965289    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdee90c-09c5-4f7c-891d-d313d806c5e3" containerName="extract"
Jul 16 17:40:14.967266 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.967249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:14.980051 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.980033    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 16 17:40:14.980957 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.980941    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zmtzb\""
Jul 16 17:40:14.981914 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.981866    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 16 17:40:14.982463 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.982437    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 16 17:40:14.982911 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.982895    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 16 17:40:14.985125 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.985109    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 16 17:40:14.993655 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:14.993630    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-7hzkd"]
Jul 16 17:40:15.060187 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.060144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d88c9190-545b-43e5-962a-d8c585744498-certificates\") pod \"keda-operator-77f648d694-7hzkd\" (UID: \"d88c9190-545b-43e5-962a-d8c585744498\") " pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:15.060377 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.060213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5x9d\" (UniqueName: \"kubernetes.io/projected/d88c9190-545b-43e5-962a-d8c585744498-kube-api-access-m5x9d\") pod \"keda-operator-77f648d694-7hzkd\" (UID: \"d88c9190-545b-43e5-962a-d8c585744498\") " pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:15.060377 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.060253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d88c9190-545b-43e5-962a-d8c585744498-cabundle0\") pod \"keda-operator-77f648d694-7hzkd\" (UID: \"d88c9190-545b-43e5-962a-d8c585744498\") " pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:15.161520 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.161484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d88c9190-545b-43e5-962a-d8c585744498-cabundle0\") pod \"keda-operator-77f648d694-7hzkd\" (UID: \"d88c9190-545b-43e5-962a-d8c585744498\") " pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:15.161717 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.161543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d88c9190-545b-43e5-962a-d8c585744498-certificates\") pod \"keda-operator-77f648d694-7hzkd\" (UID: \"d88c9190-545b-43e5-962a-d8c585744498\") " pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:15.161717 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.161611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5x9d\" (UniqueName: \"kubernetes.io/projected/d88c9190-545b-43e5-962a-d8c585744498-kube-api-access-m5x9d\") pod \"keda-operator-77f648d694-7hzkd\" (UID: \"d88c9190-545b-43e5-962a-d8c585744498\") " pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:15.161717 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:40:15.161650    2563 secret.go:281] references non-existent secret key: ca.crt
Jul 16 17:40:15.161717 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:40:15.161669    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 16 17:40:15.161717 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:40:15.161682    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-7hzkd: references non-existent secret key: ca.crt
Jul 16 17:40:15.161907 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:40:15.161750    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88c9190-545b-43e5-962a-d8c585744498-certificates podName:d88c9190-545b-43e5-962a-d8c585744498 nodeName:}" failed. No retries permitted until 2026-07-16 17:40:15.661733391 +0000 UTC m=+392.648268996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d88c9190-545b-43e5-962a-d8c585744498-certificates") pod "keda-operator-77f648d694-7hzkd" (UID: "d88c9190-545b-43e5-962a-d8c585744498") : references non-existent secret key: ca.crt
Jul 16 17:40:15.162109 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.162089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d88c9190-545b-43e5-962a-d8c585744498-cabundle0\") pod \"keda-operator-77f648d694-7hzkd\" (UID: \"d88c9190-545b-43e5-962a-d8c585744498\") " pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:15.195030 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.195002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5x9d\" (UniqueName: \"kubernetes.io/projected/d88c9190-545b-43e5-962a-d8c585744498-kube-api-access-m5x9d\") pod \"keda-operator-77f648d694-7hzkd\" (UID: \"d88c9190-545b-43e5-962a-d8c585744498\") " pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:15.665883 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.665848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d88c9190-545b-43e5-962a-d8c585744498-certificates\") pod \"keda-operator-77f648d694-7hzkd\" (UID: \"d88c9190-545b-43e5-962a-d8c585744498\") " pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:15.668728 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.668688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d88c9190-545b-43e5-962a-d8c585744498-certificates\") pod \"keda-operator-77f648d694-7hzkd\" (UID: \"d88c9190-545b-43e5-962a-d8c585744498\") " pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:15.877128 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:15.877092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:16.019339 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.019308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-7hzkd"]
Jul 16 17:40:16.022690 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:40:16.022663    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88c9190_545b_43e5_962a_d8c585744498.slice/crio-b59a32786a89b1602d15b529ec0901aff00b974dd51cd01958b0270a5b45910b WatchSource:0}: Error finding container b59a32786a89b1602d15b529ec0901aff00b974dd51cd01958b0270a5b45910b: Status 404 returned error can't find the container with id b59a32786a89b1602d15b529ec0901aff00b974dd51cd01958b0270a5b45910b
Jul 16 17:40:16.049008 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.048970    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-kvdvm"]
Jul 16 17:40:16.051824 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.051805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-kvdvm"
Jul 16 17:40:16.054941 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.054917    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 16 17:40:16.067443 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.067405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-kvdvm"]
Jul 16 17:40:16.170279 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.170178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tssc\" (UniqueName: \"kubernetes.io/projected/a67b0ac4-450b-48ea-8ea2-a8ee0f01f537-kube-api-access-6tssc\") pod \"keda-admission-56f9798bb4-kvdvm\" (UID: \"a67b0ac4-450b-48ea-8ea2-a8ee0f01f537\") " pod="openshift-keda/keda-admission-56f9798bb4-kvdvm"
Jul 16 17:40:16.170669 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.170642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a67b0ac4-450b-48ea-8ea2-a8ee0f01f537-certificates\") pod \"keda-admission-56f9798bb4-kvdvm\" (UID: \"a67b0ac4-450b-48ea-8ea2-a8ee0f01f537\") " pod="openshift-keda/keda-admission-56f9798bb4-kvdvm"
Jul 16 17:40:16.240971 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.240934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-7hzkd" event={"ID":"d88c9190-545b-43e5-962a-d8c585744498","Type":"ContainerStarted","Data":"b59a32786a89b1602d15b529ec0901aff00b974dd51cd01958b0270a5b45910b"}
Jul 16 17:40:16.271337 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.271299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a67b0ac4-450b-48ea-8ea2-a8ee0f01f537-certificates\") pod \"keda-admission-56f9798bb4-kvdvm\" (UID: \"a67b0ac4-450b-48ea-8ea2-a8ee0f01f537\") " pod="openshift-keda/keda-admission-56f9798bb4-kvdvm"
Jul 16 17:40:16.271545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.271357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tssc\" (UniqueName: \"kubernetes.io/projected/a67b0ac4-450b-48ea-8ea2-a8ee0f01f537-kube-api-access-6tssc\") pod \"keda-admission-56f9798bb4-kvdvm\" (UID: \"a67b0ac4-450b-48ea-8ea2-a8ee0f01f537\") " pod="openshift-keda/keda-admission-56f9798bb4-kvdvm"
Jul 16 17:40:16.274061 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.274032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a67b0ac4-450b-48ea-8ea2-a8ee0f01f537-certificates\") pod \"keda-admission-56f9798bb4-kvdvm\" (UID: \"a67b0ac4-450b-48ea-8ea2-a8ee0f01f537\") " pod="openshift-keda/keda-admission-56f9798bb4-kvdvm"
Jul 16 17:40:16.293253 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.293215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tssc\" (UniqueName: \"kubernetes.io/projected/a67b0ac4-450b-48ea-8ea2-a8ee0f01f537-kube-api-access-6tssc\") pod \"keda-admission-56f9798bb4-kvdvm\" (UID: \"a67b0ac4-450b-48ea-8ea2-a8ee0f01f537\") " pod="openshift-keda/keda-admission-56f9798bb4-kvdvm"
Jul 16 17:40:16.361856 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.361820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-kvdvm"
Jul 16 17:40:16.546345 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:40:16.546313    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67b0ac4_450b_48ea_8ea2_a8ee0f01f537.slice/crio-1cead06f7ffe57cef46468df8e73805a73c55113fc029f84396ee6001b773757 WatchSource:0}: Error finding container 1cead06f7ffe57cef46468df8e73805a73c55113fc029f84396ee6001b773757: Status 404 returned error can't find the container with id 1cead06f7ffe57cef46468df8e73805a73c55113fc029f84396ee6001b773757
Jul 16 17:40:16.551091 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:16.551063    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-kvdvm"]
Jul 16 17:40:17.246864 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:17.246823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-kvdvm" event={"ID":"a67b0ac4-450b-48ea-8ea2-a8ee0f01f537","Type":"ContainerStarted","Data":"1cead06f7ffe57cef46468df8e73805a73c55113fc029f84396ee6001b773757"}
Jul 16 17:40:19.256912 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:19.256859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-kvdvm" event={"ID":"a67b0ac4-450b-48ea-8ea2-a8ee0f01f537","Type":"ContainerStarted","Data":"4921313037d189cacff7c96c9db68f18155b5efb694a8bcce264c7409a2a693b"}
Jul 16 17:40:19.257314 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:19.256944    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-kvdvm"
Jul 16 17:40:19.278682 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:19.278629    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-kvdvm" podStartSLOduration=1.583898128 podStartE2EDuration="3.278607152s" totalImagesPullingTime="1.694709024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:40:19.277126612 +0000 UTC m=+396.263662221" watchObservedRunningTime="2026-07-16 17:40:19.278607152 +0000 UTC m=+396.265142759"
Jul 16 17:40:20.262520 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:20.262429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-7hzkd" event={"ID":"d88c9190-545b-43e5-962a-d8c585744498","Type":"ContainerStarted","Data":"7eade4108fd3d45806a2341cd6eb1a81e8f40b9f85ddb01246dbbd5639fee79d"}
Jul 16 17:40:20.262895 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:20.262680    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:40:20.283090 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:20.283044    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-7hzkd" podStartSLOduration=2.306341364 podStartE2EDuration="6.28302831s" totalImagesPullingTime="3.976686946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:40:20.282836217 +0000 UTC m=+397.269371823" watchObservedRunningTime="2026-07-16 17:40:20.28302831 +0000 UTC m=+397.269563915"
Jul 16 17:40:40.265164 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:40.265131    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-kvdvm"
Jul 16 17:40:41.272066 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:40:41.272035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-7hzkd"
Jul 16 17:41:28.349806 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.349774    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-jcfs9"]
Jul 16 17:41:28.356683 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.356654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-jcfs9"
Jul 16 17:41:28.363290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.363258    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 16 17:41:28.363859 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.363817    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 16 17:41:28.364094 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.364079    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 16 17:41:28.365092 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.365073    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-hz2bx\""
Jul 16 17:41:28.370630 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.370609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-jcfs9"]
Jul 16 17:41:28.457962 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.457911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8-cert\") pod \"llmisvc-controller-manager-654c557856-jcfs9\" (UID: \"bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8\") " pod="kserve/llmisvc-controller-manager-654c557856-jcfs9"
Jul 16 17:41:28.458145 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.457993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ddtq\" (UniqueName: \"kubernetes.io/projected/bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8-kube-api-access-8ddtq\") pod \"llmisvc-controller-manager-654c557856-jcfs9\" (UID: \"bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8\") " pod="kserve/llmisvc-controller-manager-654c557856-jcfs9"
Jul 16 17:41:28.558930 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.558894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8-cert\") pod \"llmisvc-controller-manager-654c557856-jcfs9\" (UID: \"bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8\") " pod="kserve/llmisvc-controller-manager-654c557856-jcfs9"
Jul 16 17:41:28.559110 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.558964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ddtq\" (UniqueName: \"kubernetes.io/projected/bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8-kube-api-access-8ddtq\") pod \"llmisvc-controller-manager-654c557856-jcfs9\" (UID: \"bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8\") " pod="kserve/llmisvc-controller-manager-654c557856-jcfs9"
Jul 16 17:41:28.561405 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.561381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8-cert\") pod \"llmisvc-controller-manager-654c557856-jcfs9\" (UID: \"bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8\") " pod="kserve/llmisvc-controller-manager-654c557856-jcfs9"
Jul 16 17:41:28.574011 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.573980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ddtq\" (UniqueName: \"kubernetes.io/projected/bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8-kube-api-access-8ddtq\") pod \"llmisvc-controller-manager-654c557856-jcfs9\" (UID: \"bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8\") " pod="kserve/llmisvc-controller-manager-654c557856-jcfs9"
Jul 16 17:41:28.667154 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.667060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-jcfs9"
Jul 16 17:41:28.825253 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:28.825228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-jcfs9"]
Jul 16 17:41:28.827351 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:41:28.827319    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbcc8784c_c8c8_427f_8af5_5d3f9ca28dd8.slice/crio-ef26cccd18ef1a39a6302cfe2127fbdf9235f62093876292530569dba9fb9050 WatchSource:0}: Error finding container ef26cccd18ef1a39a6302cfe2127fbdf9235f62093876292530569dba9fb9050: Status 404 returned error can't find the container with id ef26cccd18ef1a39a6302cfe2127fbdf9235f62093876292530569dba9fb9050
Jul 16 17:41:29.538318 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:29.538276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-jcfs9" event={"ID":"bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8","Type":"ContainerStarted","Data":"ef26cccd18ef1a39a6302cfe2127fbdf9235f62093876292530569dba9fb9050"}
Jul 16 17:41:32.552323 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:32.552285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-jcfs9" event={"ID":"bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8","Type":"ContainerStarted","Data":"f2c6301b71d21d2e2a788bf76668c6a288d55c119fbf7fa78bab93ac9bcd5b35"}
Jul 16 17:41:32.552764 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:32.552411    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-jcfs9"
Jul 16 17:41:32.575044 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:41:32.574989    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-jcfs9" podStartSLOduration=1.573709582 podStartE2EDuration="4.574971616s" totalImagesPullingTime="3.001262034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:32.574187875 +0000 UTC m=+469.560723481" watchObservedRunningTime="2026-07-16 17:41:32.574971616 +0000 UTC m=+469.561507222"
Jul 16 17:42:03.557995 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:42:03.557964    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-jcfs9"
Jul 16 17:43:43.515855 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:43:43.515821    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:43:43.517587 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:43:43.517539    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:43:43.518673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:43:43.518652    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:43:43.520304 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:43:43.520284    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:43:43.522623 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:43:43.522605    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:43:43.524151 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:43:43.524135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:44:35.967170 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:35.967134    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"]
Jul 16 17:44:35.969814 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:35.969798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:35.972471 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:35.972448    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ft4dd\""
Jul 16 17:44:35.972643 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:35.972492    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:44:35.972643 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:35.972522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-13951-kube-rbac-proxy-sar-config\""
Jul 16 17:44:35.972773 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:35.972660    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:44:35.973862 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:35.973846    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:44:35.973915 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:35.973873    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-13951-predictor-serving-cert\""
Jul 16 17:44:35.973915 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:35.973899    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 17:44:35.981162 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:35.981141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"]
Jul 16 17:44:36.079206 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.079172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.079380 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.079239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a87488-7dda-494f-a449-6f6cc6f80de9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.079380 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.079284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j66jf\" (UniqueName: \"kubernetes.io/projected/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-api-access-j66jf\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.079380 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.079313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a87488-7dda-494f-a449-6f6cc6f80de9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.079380 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.079333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.180279 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.180225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.180279 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.180285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a87488-7dda-494f-a449-6f6cc6f80de9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.180607 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.180325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j66jf\" (UniqueName: \"kubernetes.io/projected/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-api-access-j66jf\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.180607 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.180356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a87488-7dda-494f-a449-6f6cc6f80de9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.180607 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.180380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.180912 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.180880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a87488-7dda-494f-a449-6f6cc6f80de9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.181095 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.181075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.181158 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.181093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.182785 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.182765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a87488-7dda-494f-a449-6f6cc6f80de9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.189911 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.189888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j66jf\" (UniqueName: \"kubernetes.io/projected/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-api-access-j66jf\") pod \"isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.280747 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.280664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:44:36.413052 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:36.413026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"]
Jul 16 17:44:36.415287 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:44:36.415253    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a87488_7dda_494f_a449_6f6cc6f80de9.slice/crio-44a8e0f4b1be628d5f2c10008c67bef55672fc66e38c7e8a937ebae06c0d92f0 WatchSource:0}: Error finding container 44a8e0f4b1be628d5f2c10008c67bef55672fc66e38c7e8a937ebae06c0d92f0: Status 404 returned error can't find the container with id 44a8e0f4b1be628d5f2c10008c67bef55672fc66e38c7e8a937ebae06c0d92f0
Jul 16 17:44:37.282005 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:37.281964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" event={"ID":"27a87488-7dda-494f-a449-6f6cc6f80de9","Type":"ContainerStarted","Data":"44a8e0f4b1be628d5f2c10008c67bef55672fc66e38c7e8a937ebae06c0d92f0"}
Jul 16 17:44:40.299778 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:40.299737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" event={"ID":"27a87488-7dda-494f-a449-6f6cc6f80de9","Type":"ContainerStarted","Data":"1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e"}
Jul 16 17:44:44.317465 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:44.317430    2563 generic.go:356] "Generic (PLEG): container finished" podID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerID="1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e" exitCode=0
Jul 16 17:44:44.317986 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:44.317477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" event={"ID":"27a87488-7dda-494f-a449-6f6cc6f80de9","Type":"ContainerDied","Data":"1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e"}
Jul 16 17:44:44.318747 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:44.318731    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:44:59.396712 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:44:59.396615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" event={"ID":"27a87488-7dda-494f-a449-6f6cc6f80de9","Type":"ContainerStarted","Data":"90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0"}
Jul 16 17:45:02.412899 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:02.412855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" event={"ID":"27a87488-7dda-494f-a449-6f6cc6f80de9","Type":"ContainerStarted","Data":"46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e"}
Jul 16 17:45:04.424999 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:04.424963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" event={"ID":"27a87488-7dda-494f-a449-6f6cc6f80de9","Type":"ContainerStarted","Data":"2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b"}
Jul 16 17:45:04.425424 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:04.425239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:45:04.425424 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:04.425362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:45:04.426535 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:04.426505    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:45:04.448649 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:04.448602    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podStartSLOduration=2.551357621 podStartE2EDuration="29.448544836s" totalImagesPullingTime="23.105659769s" totalInitContainerRuntime="3.791527446s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:44:35 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:04.446688866 +0000 UTC m=+681.433224473" watchObservedRunningTime="2026-07-16 17:45:04.448544836 +0000 UTC m=+681.435080441"
Jul 16 17:45:05.429434 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:05.429386    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:45:05.429434 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:05.429426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:45:05.430608 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:05.430581    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:45:06.433157 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:06.433118    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:45:06.433669 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:06.433530    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:45:06.437631 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:06.437610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:45:07.437343 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:07.437301    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:45:07.437830 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:07.437600    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:45:17.438154 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:17.438114    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:45:17.438747 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:17.438623    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:45:27.437672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:27.437628    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:45:27.438151 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:27.438127    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:45:37.438157 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:37.438108    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:45:37.438736 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:37.438588    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:45:47.437925 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:47.437858    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:45:47.438375 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:47.438350    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:45:57.437476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:57.437419    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:45:57.437973 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:45:57.437952    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:46:07.437310 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:07.437243    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:46:07.437755 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:07.437670    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:46:17.438254 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:17.438211    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:46:17.438743 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:17.438311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:46:31.056710 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.056670    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"]
Jul 16 17:46:31.057174 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.057020    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" containerID="cri-o://90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0" gracePeriod=30
Jul 16 17:46:31.057174 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.057090    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kube-rbac-proxy" containerID="cri-o://46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e" gracePeriod=30
Jul 16 17:46:31.057174 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.057085    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" containerID="cri-o://2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b" gracePeriod=30
Jul 16 17:46:31.164869 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.164829    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"]
Jul 16 17:46:31.167541 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.167525    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.170398 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.170323    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-1404c-predictor-serving-cert\""
Jul 16 17:46:31.170619 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.170590    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-1404c-kube-rbac-proxy-sar-config\""
Jul 16 17:46:31.179177 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.179100    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"]
Jul 16 17:46:31.252686 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.252650    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"]
Jul 16 17:46:31.255264 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.255241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.258248 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.258228    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-1404c-predictor-serving-cert\""
Jul 16 17:46:31.258362 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.258245    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-1404c-kube-rbac-proxy-sar-config\""
Jul 16 17:46:31.267046 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.267022    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"]
Jul 16 17:46:31.361600 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.361493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186e8cfd-e44b-4914-8c08-6c598764659b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.361600 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.361540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-299tr\" (UniqueName: \"kubernetes.io/projected/186e8cfd-e44b-4914-8c08-6c598764659b-kube-api-access-299tr\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.361600 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.361599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mzg\" (UniqueName: \"kubernetes.io/projected/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-api-access-j5mzg\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.361874 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.361634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.361874 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.361666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.361874 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.361685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.361874 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.361714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186e8cfd-e44b-4914-8c08-6c598764659b-proxy-tls\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.361874 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.361783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-proxy-tls\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.361874 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.361838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.362061 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.361917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.434185 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.434144    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 16 17:46:31.463319 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186e8cfd-e44b-4914-8c08-6c598764659b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.463319 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-299tr\" (UniqueName: \"kubernetes.io/projected/186e8cfd-e44b-4914-8c08-6c598764659b-kube-api-access-299tr\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.463537 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mzg\" (UniqueName: \"kubernetes.io/projected/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-api-access-j5mzg\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.463612 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.463672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.463724 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.463724 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186e8cfd-e44b-4914-8c08-6c598764659b-proxy-tls\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.463820 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-proxy-tls\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.463820 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.463919 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.463919 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:46:31.463856    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-serving-cert: secret "isvc-xgboost-graph-raw-1404c-predictor-serving-cert" not found
Jul 16 17:46:31.464010 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:46:31.463964    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186e8cfd-e44b-4914-8c08-6c598764659b-proxy-tls podName:186e8cfd-e44b-4914-8c08-6c598764659b nodeName:}" failed. No retries permitted until 2026-07-16 17:46:31.963917005 +0000 UTC m=+768.950452608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/186e8cfd-e44b-4914-8c08-6c598764659b-proxy-tls") pod "isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" (UID: "186e8cfd-e44b-4914-8c08-6c598764659b") : secret "isvc-xgboost-graph-raw-1404c-predictor-serving-cert" not found
Jul 16 17:46:31.464154 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.464129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.464250 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.463763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186e8cfd-e44b-4914-8c08-6c598764659b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.464400 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.464372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.464526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.464440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.464526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.464442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.464628 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.464529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.466331 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.466310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-proxy-tls\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.474647 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.474622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mzg\" (UniqueName: \"kubernetes.io/projected/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-api-access-j5mzg\") pod \"isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.474766 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.474746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-299tr\" (UniqueName: \"kubernetes.io/projected/186e8cfd-e44b-4914-8c08-6c598764659b-kube-api-access-299tr\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.480539 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.480442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:31.622316 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.622289    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"]
Jul 16 17:46:31.624936 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:46:31.624907    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91b1aca_21ae_4cf9_83ae_d672d41d7d9b.slice/crio-cc3ecd1709293abba4a903e1f5a43ac780a3901ff44b946d8505fad4dcdedb22 WatchSource:0}: Error finding container cc3ecd1709293abba4a903e1f5a43ac780a3901ff44b946d8505fad4dcdedb22: Status 404 returned error can't find the container with id cc3ecd1709293abba4a903e1f5a43ac780a3901ff44b946d8505fad4dcdedb22
Jul 16 17:46:31.794488 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.794456    2563 generic.go:356] "Generic (PLEG): container finished" podID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerID="46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e" exitCode=2
Jul 16 17:46:31.794677 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.794537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" event={"ID":"27a87488-7dda-494f-a449-6f6cc6f80de9","Type":"ContainerDied","Data":"46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e"}
Jul 16 17:46:31.796085 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.796055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" event={"ID":"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b","Type":"ContainerStarted","Data":"7249eb242f47a6a74776db976f1271f58ab4b06079eccb503a27b022c272ce86"}
Jul 16 17:46:31.796196 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.796112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" event={"ID":"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b","Type":"ContainerStarted","Data":"cc3ecd1709293abba4a903e1f5a43ac780a3901ff44b946d8505fad4dcdedb22"}
Jul 16 17:46:31.966933 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.966840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186e8cfd-e44b-4914-8c08-6c598764659b-proxy-tls\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:31.969216 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:31.969196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186e8cfd-e44b-4914-8c08-6c598764659b-proxy-tls\") pod \"isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:32.165347 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:32.165294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:32.304178 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:32.304151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"]
Jul 16 17:46:32.306972 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:46:32.306942    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186e8cfd_e44b_4914_8c08_6c598764659b.slice/crio-14426625a6701dccf5ac692fac91a1d4abdf1b152cafea99251dce9a795e7f39 WatchSource:0}: Error finding container 14426625a6701dccf5ac692fac91a1d4abdf1b152cafea99251dce9a795e7f39: Status 404 returned error can't find the container with id 14426625a6701dccf5ac692fac91a1d4abdf1b152cafea99251dce9a795e7f39
Jul 16 17:46:32.803627 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:32.803583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" event={"ID":"186e8cfd-e44b-4914-8c08-6c598764659b","Type":"ContainerStarted","Data":"8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c"}
Jul 16 17:46:32.803824 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:32.803660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" event={"ID":"186e8cfd-e44b-4914-8c08-6c598764659b","Type":"ContainerStarted","Data":"14426625a6701dccf5ac692fac91a1d4abdf1b152cafea99251dce9a795e7f39"}
Jul 16 17:46:35.820840 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:35.820802    2563 generic.go:356] "Generic (PLEG): container finished" podID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerID="7249eb242f47a6a74776db976f1271f58ab4b06079eccb503a27b022c272ce86" exitCode=0
Jul 16 17:46:35.821350 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:35.820850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" event={"ID":"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b","Type":"ContainerDied","Data":"7249eb242f47a6a74776db976f1271f58ab4b06079eccb503a27b022c272ce86"}
Jul 16 17:46:36.433792 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:36.433731    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 16 17:46:36.827350 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:36.827315    2563 generic.go:356] "Generic (PLEG): container finished" podID="186e8cfd-e44b-4914-8c08-6c598764659b" containerID="8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c" exitCode=0
Jul 16 17:46:36.827823 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:36.827384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" event={"ID":"186e8cfd-e44b-4914-8c08-6c598764659b","Type":"ContainerDied","Data":"8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c"}
Jul 16 17:46:36.830024 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:36.830003    2563 generic.go:356] "Generic (PLEG): container finished" podID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerID="90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0" exitCode=0
Jul 16 17:46:36.830108 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:36.830072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" event={"ID":"27a87488-7dda-494f-a449-6f6cc6f80de9","Type":"ContainerDied","Data":"90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0"}
Jul 16 17:46:36.832014 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:36.831995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" event={"ID":"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b","Type":"ContainerStarted","Data":"229114b736e7ca20cd7364db1deee4cd6802e8f304f32fedafc22792668160df"}
Jul 16 17:46:36.832091 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:36.832044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" event={"ID":"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b","Type":"ContainerStarted","Data":"c4fae96ae0e92bcc829733f9543b73d089807dd3eecfed6382a09ab0b8b66b8e"}
Jul 16 17:46:36.832272 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:36.832257    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:36.867822 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:36.867766    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podStartSLOduration=2.115450719 podStartE2EDuration="5.867747264s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.752296545s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:36.86650101 +0000 UTC m=+773.853036615" watchObservedRunningTime="2026-07-16 17:46:36.867747264 +0000 UTC m=+773.854282933"
Jul 16 17:46:37.437379 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:37.437307    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:46:37.437772 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:37.437745    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:46:37.837805 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:37.837756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:37.839168 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:37.839134    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 16 17:46:38.844682 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:38.844642    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 16 17:46:41.433932 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:41.433891    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 16 17:46:41.434351 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:41.434042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:46:43.849501 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:43.849461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:46:43.850133 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:43.850102    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 16 17:46:46.433934 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:46.433883    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 16 17:46:47.437828 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:47.437777    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:46:47.438309 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:47.438111    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:46:51.434013 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:51.433969    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 16 17:46:53.850857 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:53.850814    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 16 17:46:56.434143 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:56.434094    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 16 17:46:56.940834 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:56.940798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" event={"ID":"186e8cfd-e44b-4914-8c08-6c598764659b","Type":"ContainerStarted","Data":"e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c"}
Jul 16 17:46:56.941007 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:56.940860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" event={"ID":"186e8cfd-e44b-4914-8c08-6c598764659b","Type":"ContainerStarted","Data":"bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713"}
Jul 16 17:46:56.941069 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:56.941049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:56.961308 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:56.961232    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podStartSLOduration=2.164625082 podStartE2EDuration="25.961213599s" totalImagesPullingTime="19.94365873s" totalInitContainerRuntime="3.852929787s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:56.96098863 +0000 UTC m=+793.947524248" watchObservedRunningTime="2026-07-16 17:46:56.961213599 +0000 UTC m=+793.947749209"
Jul 16 17:46:57.438200 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:57.438148    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 16 17:46:57.438699 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:57.438304    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:46:57.438699 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:57.438499    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:46:57.438699 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:57.438629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:46:57.945864 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:57.945832    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:46:57.947341 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:57.947309    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 16 17:46:58.950574 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:46:58.950521    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 16 17:47:01.214706 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.214677    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:47:01.321083 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.320991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j66jf\" (UniqueName: \"kubernetes.io/projected/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-api-access-j66jf\") pod \"27a87488-7dda-494f-a449-6f6cc6f80de9\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") "
Jul 16 17:47:01.321083 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.321073    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a87488-7dda-494f-a449-6f6cc6f80de9-proxy-tls\") pod \"27a87488-7dda-494f-a449-6f6cc6f80de9\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") "
Jul 16 17:47:01.321266 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.321244    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a87488-7dda-494f-a449-6f6cc6f80de9-kserve-provision-location\") pod \"27a87488-7dda-494f-a449-6f6cc6f80de9\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") "
Jul 16 17:47:01.321358 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.321297    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-cabundle-cert\") pod \"27a87488-7dda-494f-a449-6f6cc6f80de9\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") "
Jul 16 17:47:01.321490 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.321363    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-rbac-proxy-sar-config\") pod \"27a87488-7dda-494f-a449-6f6cc6f80de9\" (UID: \"27a87488-7dda-494f-a449-6f6cc6f80de9\") "
Jul 16 17:47:01.321643 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.321613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a87488-7dda-494f-a449-6f6cc6f80de9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27a87488-7dda-494f-a449-6f6cc6f80de9" (UID: "27a87488-7dda-494f-a449-6f6cc6f80de9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:47:01.321803 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.321786    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a87488-7dda-494f-a449-6f6cc6f80de9-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:47:01.321865 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.321780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27a87488-7dda-494f-a449-6f6cc6f80de9" (UID: "27a87488-7dda-494f-a449-6f6cc6f80de9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:47:01.321865 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.321819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27a87488-7dda-494f-a449-6f6cc6f80de9" (UID: "27a87488-7dda-494f-a449-6f6cc6f80de9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:47:01.323292 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.323265    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a87488-7dda-494f-a449-6f6cc6f80de9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27a87488-7dda-494f-a449-6f6cc6f80de9" (UID: "27a87488-7dda-494f-a449-6f6cc6f80de9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:47:01.323392 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.323298    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-api-access-j66jf" (OuterVolumeSpecName: "kube-api-access-j66jf") pod "27a87488-7dda-494f-a449-6f6cc6f80de9" (UID: "27a87488-7dda-494f-a449-6f6cc6f80de9"). InnerVolumeSpecName "kube-api-access-j66jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:47:01.422476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.422427    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j66jf\" (UniqueName: \"kubernetes.io/projected/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-api-access-j66jf\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:47:01.422476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.422460    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a87488-7dda-494f-a449-6f6cc6f80de9-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:47:01.422476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.422470    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:47:01.422476 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.422479    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27a87488-7dda-494f-a449-6f6cc6f80de9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:47:01.966011 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.965909    2563 generic.go:356] "Generic (PLEG): container finished" podID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerID="2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b" exitCode=0
Jul 16 17:47:01.966011 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.965956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" event={"ID":"27a87488-7dda-494f-a449-6f6cc6f80de9","Type":"ContainerDied","Data":"2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b"}
Jul 16 17:47:01.966011 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.965979    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"
Jul 16 17:47:01.966011 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.966010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln" event={"ID":"27a87488-7dda-494f-a449-6f6cc6f80de9","Type":"ContainerDied","Data":"44a8e0f4b1be628d5f2c10008c67bef55672fc66e38c7e8a937ebae06c0d92f0"}
Jul 16 17:47:01.966305 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.966042    2563 scope.go:117] "RemoveContainer" containerID="2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b"
Jul 16 17:47:01.974021 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.974004    2563 scope.go:117] "RemoveContainer" containerID="46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e"
Jul 16 17:47:01.981774 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.981753    2563 scope.go:117] "RemoveContainer" containerID="90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0"
Jul 16 17:47:01.985985 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.985964    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"]
Jul 16 17:47:01.988938 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.988915    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13951-predictor-9454757dd-v4sln"]
Jul 16 17:47:01.990438 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.990416    2563 scope.go:117] "RemoveContainer" containerID="1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e"
Jul 16 17:47:01.998275 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.998252    2563 scope.go:117] "RemoveContainer" containerID="2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b"
Jul 16 17:47:01.998597 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:47:01.998576    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b\": container with ID starting with 2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b not found: ID does not exist" containerID="2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b"
Jul 16 17:47:01.998666 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.998606    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b"} err="failed to get container status \"2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b\": rpc error: code = NotFound desc = could not find container \"2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b\": container with ID starting with 2295df6516adda8b557eabd93cb9c054c3faf2f02c43e917df21039871da605b not found: ID does not exist"
Jul 16 17:47:01.998666 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.998638    2563 scope.go:117] "RemoveContainer" containerID="46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e"
Jul 16 17:47:01.998913 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:47:01.998896    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e\": container with ID starting with 46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e not found: ID does not exist" containerID="46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e"
Jul 16 17:47:01.998975 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.998922    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e"} err="failed to get container status \"46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e\": rpc error: code = NotFound desc = could not find container \"46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e\": container with ID starting with 46a6d36e3cae8fbe5eee7be1a68503b24b2c92a5a268c4bc31b3880302c9829e not found: ID does not exist"
Jul 16 17:47:01.998975 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.998947    2563 scope.go:117] "RemoveContainer" containerID="90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0"
Jul 16 17:47:01.999231 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:47:01.999214    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0\": container with ID starting with 90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0 not found: ID does not exist" containerID="90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0"
Jul 16 17:47:01.999318 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.999236    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0"} err="failed to get container status \"90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0\": rpc error: code = NotFound desc = could not find container \"90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0\": container with ID starting with 90b8c5ba1c0911d275c396c3c2ecf3399af6ef0199949351e9202ec0322f86f0 not found: ID does not exist"
Jul 16 17:47:01.999318 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.999251    2563 scope.go:117] "RemoveContainer" containerID="1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e"
Jul 16 17:47:01.999502 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:47:01.999486    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e\": container with ID starting with 1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e not found: ID does not exist" containerID="1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e"
Jul 16 17:47:01.999576 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:01.999506    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e"} err="failed to get container status \"1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e\": rpc error: code = NotFound desc = could not find container \"1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e\": container with ID starting with 1287d87686bf8e497419852624f49c9e68030eae55b5f75ae297f36c31d3da5e not found: ID does not exist"
Jul 16 17:47:03.677536 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:03.677502    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" path="/var/lib/kubelet/pods/27a87488-7dda-494f-a449-6f6cc6f80de9/volumes"
Jul 16 17:47:03.850129 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:03.850091    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 16 17:47:03.956629 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:03.956536    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:47:03.957082 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:03.957054    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 16 17:47:13.850994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:13.850911    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 16 17:47:13.957357 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:13.957317    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 16 17:47:23.849978 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:23.849940    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 16 17:47:23.957643 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:23.957604    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 16 17:47:33.850475 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:33.850430    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 16 17:47:33.957624 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:33.957578    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 16 17:47:43.850822 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:43.850722    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 16 17:47:43.957227 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:43.957185    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 16 17:47:53.853660 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:53.853630    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:47:53.957673 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:47:53.957634    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 16 17:48:03.957706 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:03.957675    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:48:21.410790 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.410754    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"]
Jul 16 17:48:21.411206 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.411003    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" containerID="cri-o://c4fae96ae0e92bcc829733f9543b73d089807dd3eecfed6382a09ab0b8b66b8e" gracePeriod=30
Jul 16 17:48:21.411206 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.411062    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kube-rbac-proxy" containerID="cri-o://229114b736e7ca20cd7364db1deee4cd6802e8f304f32fedafc22792668160df" gracePeriod=30
Jul 16 17:48:21.524982 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.524928    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"]
Jul 16 17:48:21.525343 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525327    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kube-rbac-proxy"
Jul 16 17:48:21.525343 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525343    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kube-rbac-proxy"
Jul 16 17:48:21.525459 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525367    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="storage-initializer"
Jul 16 17:48:21.525459 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525373    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="storage-initializer"
Jul 16 17:48:21.525459 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525380    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container"
Jul 16 17:48:21.525459 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525387    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container"
Jul 16 17:48:21.525459 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525395    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent"
Jul 16 17:48:21.525459 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525401    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent"
Jul 16 17:48:21.525707 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525466    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kube-rbac-proxy"
Jul 16 17:48:21.525707 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525476    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="agent"
Jul 16 17:48:21.525707 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.525482    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a87488-7dda-494f-a449-6f6cc6f80de9" containerName="kserve-container"
Jul 16 17:48:21.533448 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.533420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.536886 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.536861    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-57df2-kube-rbac-proxy-sar-config\""
Jul 16 17:48:21.537027 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.536862    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-57df2-predictor-serving-cert\""
Jul 16 17:48:21.545710 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.545685    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"]
Jul 16 17:48:21.694906 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.694812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/580d18b7-efd9-410c-acda-0e237b5941d8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.694906 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.694856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.694906 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.694894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.695121 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.694961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-497zk\" (UniqueName: \"kubernetes.io/projected/580d18b7-efd9-410c-acda-0e237b5941d8-kube-api-access-497zk\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.695121 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.695007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/580d18b7-efd9-410c-acda-0e237b5941d8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.796436 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.796390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/580d18b7-efd9-410c-acda-0e237b5941d8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.796638 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.796477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/580d18b7-efd9-410c-acda-0e237b5941d8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.796638 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.796523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.796638 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.796581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.796794 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.796640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-497zk\" (UniqueName: \"kubernetes.io/projected/580d18b7-efd9-410c-acda-0e237b5941d8-kube-api-access-497zk\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.796922 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.796895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/580d18b7-efd9-410c-acda-0e237b5941d8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.797332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.797306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.797332 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.797316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.799579 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.799534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/580d18b7-efd9-410c-acda-0e237b5941d8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.817840 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.817801    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"]
Jul 16 17:48:21.817979 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.817886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-497zk\" (UniqueName: \"kubernetes.io/projected/580d18b7-efd9-410c-acda-0e237b5941d8-kube-api-access-497zk\") pod \"isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.818086 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.818046    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container" containerID="cri-o://bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713" gracePeriod=30
Jul 16 17:48:21.818156 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.818105    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kube-rbac-proxy" containerID="cri-o://e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c" gracePeriod=30
Jul 16 17:48:21.843675 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.843635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:21.983641 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:21.983573    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"]
Jul 16 17:48:21.985913 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:48:21.985884    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580d18b7_efd9_410c_acda_0e237b5941d8.slice/crio-d5c9410f68d8377ef3a26e1b4b2cdfe07f6532a5c0c65243848059b1c4a86e05 WatchSource:0}: Error finding container d5c9410f68d8377ef3a26e1b4b2cdfe07f6532a5c0c65243848059b1c4a86e05: Status 404 returned error can't find the container with id d5c9410f68d8377ef3a26e1b4b2cdfe07f6532a5c0c65243848059b1c4a86e05
Jul 16 17:48:22.335786 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:22.335701    2563 generic.go:356] "Generic (PLEG): container finished" podID="186e8cfd-e44b-4914-8c08-6c598764659b" containerID="e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c" exitCode=2
Jul 16 17:48:22.335932 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:22.335776    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" event={"ID":"186e8cfd-e44b-4914-8c08-6c598764659b","Type":"ContainerDied","Data":"e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c"}
Jul 16 17:48:22.337392 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:22.337361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" event={"ID":"580d18b7-efd9-410c-acda-0e237b5941d8","Type":"ContainerStarted","Data":"f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57"}
Jul 16 17:48:22.337523 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:22.337414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" event={"ID":"580d18b7-efd9-410c-acda-0e237b5941d8","Type":"ContainerStarted","Data":"d5c9410f68d8377ef3a26e1b4b2cdfe07f6532a5c0c65243848059b1c4a86e05"}
Jul 16 17:48:22.339510 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:22.339490    2563 generic.go:356] "Generic (PLEG): container finished" podID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerID="229114b736e7ca20cd7364db1deee4cd6802e8f304f32fedafc22792668160df" exitCode=2
Jul 16 17:48:22.339603 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:22.339520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" event={"ID":"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b","Type":"ContainerDied","Data":"229114b736e7ca20cd7364db1deee4cd6802e8f304f32fedafc22792668160df"}
Jul 16 17:48:23.844650 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:23.844604    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 16 17:48:23.851007 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:23.850962    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 16 17:48:23.951245 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:23.951206    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 16 17:48:23.957042 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:23.957005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 16 17:48:25.672940 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.672908    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:48:25.835656 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.835535    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-299tr\" (UniqueName: \"kubernetes.io/projected/186e8cfd-e44b-4914-8c08-6c598764659b-kube-api-access-299tr\") pod \"186e8cfd-e44b-4914-8c08-6c598764659b\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") "
Jul 16 17:48:25.835656 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.835629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-cabundle-cert\") pod \"186e8cfd-e44b-4914-8c08-6c598764659b\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") "
Jul 16 17:48:25.835865 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.835684    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186e8cfd-e44b-4914-8c08-6c598764659b-proxy-tls\") pod \"186e8cfd-e44b-4914-8c08-6c598764659b\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") "
Jul 16 17:48:25.835865 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.835740    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186e8cfd-e44b-4914-8c08-6c598764659b-kserve-provision-location\") pod \"186e8cfd-e44b-4914-8c08-6c598764659b\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") "
Jul 16 17:48:25.835865 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.835772    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-kube-rbac-proxy-sar-config\") pod \"186e8cfd-e44b-4914-8c08-6c598764659b\" (UID: \"186e8cfd-e44b-4914-8c08-6c598764659b\") "
Jul 16 17:48:25.836104 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.836070    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186e8cfd-e44b-4914-8c08-6c598764659b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "186e8cfd-e44b-4914-8c08-6c598764659b" (UID: "186e8cfd-e44b-4914-8c08-6c598764659b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:25.836221 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.836108    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "186e8cfd-e44b-4914-8c08-6c598764659b" (UID: "186e8cfd-e44b-4914-8c08-6c598764659b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:48:25.836221 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.836154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "186e8cfd-e44b-4914-8c08-6c598764659b" (UID: "186e8cfd-e44b-4914-8c08-6c598764659b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:48:25.837834 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.837806    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186e8cfd-e44b-4914-8c08-6c598764659b-kube-api-access-299tr" (OuterVolumeSpecName: "kube-api-access-299tr") pod "186e8cfd-e44b-4914-8c08-6c598764659b" (UID: "186e8cfd-e44b-4914-8c08-6c598764659b"). InnerVolumeSpecName "kube-api-access-299tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:48:25.837834 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.837824    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186e8cfd-e44b-4914-8c08-6c598764659b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "186e8cfd-e44b-4914-8c08-6c598764659b" (UID: "186e8cfd-e44b-4914-8c08-6c598764659b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:48:25.937047 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.937008    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-299tr\" (UniqueName: \"kubernetes.io/projected/186e8cfd-e44b-4914-8c08-6c598764659b-kube-api-access-299tr\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:48:25.937047 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.937040    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:48:25.937047 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.937052    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186e8cfd-e44b-4914-8c08-6c598764659b-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:48:25.937272 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.937061    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186e8cfd-e44b-4914-8c08-6c598764659b-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:48:25.937272 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:25.937073    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186e8cfd-e44b-4914-8c08-6c598764659b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:48:26.365224 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.365167    2563 generic.go:356] "Generic (PLEG): container finished" podID="580d18b7-efd9-410c-acda-0e237b5941d8" containerID="f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57" exitCode=0
Jul 16 17:48:26.365411 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.365296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" event={"ID":"580d18b7-efd9-410c-acda-0e237b5941d8","Type":"ContainerDied","Data":"f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57"}
Jul 16 17:48:26.367817 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.367788    2563 generic.go:356] "Generic (PLEG): container finished" podID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerID="c4fae96ae0e92bcc829733f9543b73d089807dd3eecfed6382a09ab0b8b66b8e" exitCode=0
Jul 16 17:48:26.367915 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.367866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" event={"ID":"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b","Type":"ContainerDied","Data":"c4fae96ae0e92bcc829733f9543b73d089807dd3eecfed6382a09ab0b8b66b8e"}
Jul 16 17:48:26.369928 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.369879    2563 generic.go:356] "Generic (PLEG): container finished" podID="186e8cfd-e44b-4914-8c08-6c598764659b" containerID="bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713" exitCode=0
Jul 16 17:48:26.369928 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.369912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" event={"ID":"186e8cfd-e44b-4914-8c08-6c598764659b","Type":"ContainerDied","Data":"bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713"}
Jul 16 17:48:26.370087 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.369952    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"
Jul 16 17:48:26.370087 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.369967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb" event={"ID":"186e8cfd-e44b-4914-8c08-6c598764659b","Type":"ContainerDied","Data":"14426625a6701dccf5ac692fac91a1d4abdf1b152cafea99251dce9a795e7f39"}
Jul 16 17:48:26.370087 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.370017    2563 scope.go:117] "RemoveContainer" containerID="e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c"
Jul 16 17:48:26.383095 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.383076    2563 scope.go:117] "RemoveContainer" containerID="bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713"
Jul 16 17:48:26.393776 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.393709    2563 scope.go:117] "RemoveContainer" containerID="8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c"
Jul 16 17:48:26.401912 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.401888    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"]
Jul 16 17:48:26.407034 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.406989    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1404c-predictor-857fcf848f-fnmjb"]
Jul 16 17:48:26.409602 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.409498    2563 scope.go:117] "RemoveContainer" containerID="e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c"
Jul 16 17:48:26.410338 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:48:26.410080    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c\": container with ID starting with e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c not found: ID does not exist" containerID="e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c"
Jul 16 17:48:26.410338 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.410122    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c"} err="failed to get container status \"e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c\": rpc error: code = NotFound desc = could not find container \"e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c\": container with ID starting with e64d25fcaa8dd673ee809fc38f85639e107b2bb67d75f59589102ff9f41b3e9c not found: ID does not exist"
Jul 16 17:48:26.410338 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.410151    2563 scope.go:117] "RemoveContainer" containerID="bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713"
Jul 16 17:48:26.410633 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:48:26.410475    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713\": container with ID starting with bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713 not found: ID does not exist" containerID="bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713"
Jul 16 17:48:26.410633 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.410507    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713"} err="failed to get container status \"bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713\": rpc error: code = NotFound desc = could not find container \"bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713\": container with ID starting with bfa8aa96edcd7c8765a4780331675efcbf9f4de0d56f732058a5327ccd5ab713 not found: ID does not exist"
Jul 16 17:48:26.410633 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.410536    2563 scope.go:117] "RemoveContainer" containerID="8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c"
Jul 16 17:48:26.413107 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:48:26.410830    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c\": container with ID starting with 8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c not found: ID does not exist" containerID="8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c"
Jul 16 17:48:26.413107 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.410861    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c"} err="failed to get container status \"8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c\": rpc error: code = NotFound desc = could not find container \"8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c\": container with ID starting with 8d50ccd3962e4a1cc3db42700b99a29e2fb268e0deddf21d19602f37be764e7c not found: ID does not exist"
Jul 16 17:48:26.472089 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.472064    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:48:26.645861 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.645770    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-rbac-proxy-sar-config\") pod \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") "
Jul 16 17:48:26.645861 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.645812    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-proxy-tls\") pod \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") "
Jul 16 17:48:26.645861 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.645860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5mzg\" (UniqueName: \"kubernetes.io/projected/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-api-access-j5mzg\") pod \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") "
Jul 16 17:48:26.646094 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.645953    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kserve-provision-location\") pod \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") "
Jul 16 17:48:26.646094 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.646002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-cabundle-cert\") pod \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\" (UID: \"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b\") "
Jul 16 17:48:26.646267 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.646226    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" (UID: "b91b1aca-21ae-4cf9-83ae-d672d41d7d9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:26.646341 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.646234    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" (UID: "b91b1aca-21ae-4cf9-83ae-d672d41d7d9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:48:26.646414 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.646391    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" (UID: "b91b1aca-21ae-4cf9-83ae-d672d41d7d9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:48:26.647932 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.647912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" (UID: "b91b1aca-21ae-4cf9-83ae-d672d41d7d9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:48:26.648075 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.648057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-api-access-j5mzg" (OuterVolumeSpecName: "kube-api-access-j5mzg") pod "b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" (UID: "b91b1aca-21ae-4cf9-83ae-d672d41d7d9b"). InnerVolumeSpecName "kube-api-access-j5mzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:48:26.746873 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.746833    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:48:26.746873 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.746867    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:48:26.746873 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.746877    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5mzg\" (UniqueName: \"kubernetes.io/projected/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kube-api-access-j5mzg\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:48:26.746873 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.746886    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:48:26.747348 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:26.746895    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:48:27.377920 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.377885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" event={"ID":"580d18b7-efd9-410c-acda-0e237b5941d8","Type":"ContainerStarted","Data":"d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809"}
Jul 16 17:48:27.378116 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.377951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" event={"ID":"580d18b7-efd9-410c-acda-0e237b5941d8","Type":"ContainerStarted","Data":"fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878"}
Jul 16 17:48:27.378182 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.378168    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:27.379736 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.379711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v" event={"ID":"b91b1aca-21ae-4cf9-83ae-d672d41d7d9b","Type":"ContainerDied","Data":"cc3ecd1709293abba4a903e1f5a43ac780a3901ff44b946d8505fad4dcdedb22"}
Jul 16 17:48:27.379868 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.379752    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"
Jul 16 17:48:27.379868 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.379779    2563 scope.go:117] "RemoveContainer" containerID="229114b736e7ca20cd7364db1deee4cd6802e8f304f32fedafc22792668160df"
Jul 16 17:48:27.388957 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.388938    2563 scope.go:117] "RemoveContainer" containerID="c4fae96ae0e92bcc829733f9543b73d089807dd3eecfed6382a09ab0b8b66b8e"
Jul 16 17:48:27.396589 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.396545    2563 scope.go:117] "RemoveContainer" containerID="7249eb242f47a6a74776db976f1271f58ab4b06079eccb503a27b022c272ce86"
Jul 16 17:48:27.399449 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.399404    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podStartSLOduration=2.567024832 podStartE2EDuration="6.399387587s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.832362755s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:27.397761053 +0000 UTC m=+884.384296661" watchObservedRunningTime="2026-07-16 17:48:27.399387587 +0000 UTC m=+884.385923193"
Jul 16 17:48:27.412275 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.412247    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"]
Jul 16 17:48:27.418324 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.418301    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1404c-predictor-dbdf54768-4dv6v"]
Jul 16 17:48:27.673994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.673916    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" path="/var/lib/kubelet/pods/186e8cfd-e44b-4914-8c08-6c598764659b/volumes"
Jul 16 17:48:27.674407 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:27.674393    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" path="/var/lib/kubelet/pods/b91b1aca-21ae-4cf9-83ae-d672d41d7d9b/volumes"
Jul 16 17:48:28.385315 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:28.385286    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:28.386780 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:28.386754    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 16 17:48:29.389768 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:29.389734    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 16 17:48:34.394973 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:34.394943    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:48:34.395664 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:34.395632    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 16 17:48:43.542652 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:43.542623    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:48:43.545127 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:43.545103    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:48:43.545357 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:43.545337    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:48:43.548277 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:43.548246    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:48:43.549901 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:43.549876    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:48:43.552335 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:43.552317    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:48:44.396366 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:44.396331    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 16 17:48:54.395761 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:48:54.395718    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 16 17:49:04.395628 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:49:04.395579    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 16 17:49:14.396577 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:49:14.396509    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 16 17:49:24.396329 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:49:24.396286    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 16 17:49:34.395907 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:49:34.395859    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 16 17:49:44.396255 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:49:44.396223    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:50:11.811581 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.811469    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"]
Jul 16 17:50:11.812078 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.811751    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" containerID="cri-o://fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878" gracePeriod=30
Jul 16 17:50:11.812078 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.811831    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kube-rbac-proxy" containerID="cri-o://d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809" gracePeriod=30
Jul 16 17:50:11.870110 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870076    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"]
Jul 16 17:50:11.870417 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870406    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container"
Jul 16 17:50:11.870461 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870419    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container"
Jul 16 17:50:11.870461 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870427    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kube-rbac-proxy"
Jul 16 17:50:11.870461 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870432    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kube-rbac-proxy"
Jul 16 17:50:11.870461 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870439    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kube-rbac-proxy"
Jul 16 17:50:11.870461 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870444    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kube-rbac-proxy"
Jul 16 17:50:11.870461 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870462    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="storage-initializer"
Jul 16 17:50:11.870653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870468    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="storage-initializer"
Jul 16 17:50:11.870653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870475    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container"
Jul 16 17:50:11.870653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870480    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container"
Jul 16 17:50:11.870653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870490    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="storage-initializer"
Jul 16 17:50:11.870653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870495    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="storage-initializer"
Jul 16 17:50:11.870653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870545    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kube-rbac-proxy"
Jul 16 17:50:11.870653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870569    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kube-rbac-proxy"
Jul 16 17:50:11.870653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870575    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="186e8cfd-e44b-4914-8c08-6c598764659b" containerName="kserve-container"
Jul 16 17:50:11.870653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.870581    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91b1aca-21ae-4cf9-83ae-d672d41d7d9b" containerName="kserve-container"
Jul 16 17:50:11.873593 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.873575    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:11.876806 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.876780    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-c3a17-predictor-serving-cert\""
Jul 16 17:50:11.876937 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.876781    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-c3a17-kube-rbac-proxy-sar-config\""
Jul 16 17:50:11.893142 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.893111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"]
Jul 16 17:50:11.941961 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.941929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwq7x\" (UniqueName: \"kubernetes.io/projected/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-api-access-lwq7x\") pod \"message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") " pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:11.942108 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.941978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") " pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:11.942108 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:11.942038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15bc05f5-8881-495b-8510-ac0076c95f6f-proxy-tls\") pod \"message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") " pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:12.042618 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.042574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwq7x\" (UniqueName: \"kubernetes.io/projected/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-api-access-lwq7x\") pod \"message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") " pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:12.042820 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.042633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") " pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:12.042820 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.042656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15bc05f5-8881-495b-8510-ac0076c95f6f-proxy-tls\") pod \"message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") " pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:12.042820 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:50:12.042770    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-serving-cert: secret "message-dumper-raw-c3a17-predictor-serving-cert" not found
Jul 16 17:50:12.043069 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:50:12.042835    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15bc05f5-8881-495b-8510-ac0076c95f6f-proxy-tls podName:15bc05f5-8881-495b-8510-ac0076c95f6f nodeName:}" failed. No retries permitted until 2026-07-16 17:50:12.542813956 +0000 UTC m=+989.529349554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15bc05f5-8881-495b-8510-ac0076c95f6f-proxy-tls") pod "message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c" (UID: "15bc05f5-8881-495b-8510-ac0076c95f6f") : secret "message-dumper-raw-c3a17-predictor-serving-cert" not found
Jul 16 17:50:12.043388 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.043361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") " pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:12.060885 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.060853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwq7x\" (UniqueName: \"kubernetes.io/projected/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-api-access-lwq7x\") pod \"message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") " pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:12.548041 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.547992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15bc05f5-8881-495b-8510-ac0076c95f6f-proxy-tls\") pod \"message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") " pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:12.550394 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.550361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15bc05f5-8881-495b-8510-ac0076c95f6f-proxy-tls\") pod \"message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") " pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:12.783279 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.783239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:12.830839 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.830737    2563 generic.go:356] "Generic (PLEG): container finished" podID="580d18b7-efd9-410c-acda-0e237b5941d8" containerID="d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809" exitCode=2
Jul 16 17:50:12.830839 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.830818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" event={"ID":"580d18b7-efd9-410c-acda-0e237b5941d8","Type":"ContainerDied","Data":"d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809"}
Jul 16 17:50:12.921659 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.921631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"]
Jul 16 17:50:12.924332 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:50:12.924299    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bc05f5_8881_495b_8510_ac0076c95f6f.slice/crio-e976729600962b8910f24eb044149603b9077a1950c2133fee537c8097782e0b WatchSource:0}: Error finding container e976729600962b8910f24eb044149603b9077a1950c2133fee537c8097782e0b: Status 404 returned error can't find the container with id e976729600962b8910f24eb044149603b9077a1950c2133fee537c8097782e0b
Jul 16 17:50:12.926341 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:12.926318    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:50:13.836277 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:13.836240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c" event={"ID":"15bc05f5-8881-495b-8510-ac0076c95f6f","Type":"ContainerStarted","Data":"e976729600962b8910f24eb044149603b9077a1950c2133fee537c8097782e0b"}
Jul 16 17:50:14.390316 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:14.390256    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 16 17:50:14.396060 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:14.396016    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 16 17:50:14.842484 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:14.842446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c" event={"ID":"15bc05f5-8881-495b-8510-ac0076c95f6f","Type":"ContainerStarted","Data":"cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993"}
Jul 16 17:50:14.842909 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:14.842518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c" event={"ID":"15bc05f5-8881-495b-8510-ac0076c95f6f","Type":"ContainerStarted","Data":"26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de"}
Jul 16 17:50:14.842909 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:14.842580    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:14.860473 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:14.860424    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c" podStartSLOduration=2.2656701 podStartE2EDuration="3.8604066s" totalImagesPullingTime="1.5947365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:50:14.859662547 +0000 UTC m=+991.846198153" watchObservedRunningTime="2026-07-16 17:50:14.8604066 +0000 UTC m=+991.846942205"
Jul 16 17:50:15.847099 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:15.847065    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:15.848842 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:15.848816    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:16.768544 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.768518    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:50:16.852696 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.852660    2563 generic.go:356] "Generic (PLEG): container finished" podID="580d18b7-efd9-410c-acda-0e237b5941d8" containerID="fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878" exitCode=0
Jul 16 17:50:16.853165 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.852722    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"
Jul 16 17:50:16.853165 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.852739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" event={"ID":"580d18b7-efd9-410c-acda-0e237b5941d8","Type":"ContainerDied","Data":"fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878"}
Jul 16 17:50:16.853165 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.852797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb" event={"ID":"580d18b7-efd9-410c-acda-0e237b5941d8","Type":"ContainerDied","Data":"d5c9410f68d8377ef3a26e1b4b2cdfe07f6532a5c0c65243848059b1c4a86e05"}
Jul 16 17:50:16.853165 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.852825    2563 scope.go:117] "RemoveContainer" containerID="d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809"
Jul 16 17:50:16.861128 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.861111    2563 scope.go:117] "RemoveContainer" containerID="fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878"
Jul 16 17:50:16.868810 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.868788    2563 scope.go:117] "RemoveContainer" containerID="f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57"
Jul 16 17:50:16.876244 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.876223    2563 scope.go:117] "RemoveContainer" containerID="d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809"
Jul 16 17:50:16.876545 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:50:16.876523    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809\": container with ID starting with d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809 not found: ID does not exist" containerID="d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809"
Jul 16 17:50:16.876545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.876629    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809"} err="failed to get container status \"d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809\": rpc error: code = NotFound desc = could not find container \"d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809\": container with ID starting with d19c7b372da63c0c21cefdad49f58c6499310c1d0efcb66fe20d34658e454809 not found: ID does not exist"
Jul 16 17:50:16.876545 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.876660    2563 scope.go:117] "RemoveContainer" containerID="fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878"
Jul 16 17:50:16.876916 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:50:16.876897    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878\": container with ID starting with fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878 not found: ID does not exist" containerID="fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878"
Jul 16 17:50:16.876966 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.876926    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878"} err="failed to get container status \"fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878\": rpc error: code = NotFound desc = could not find container \"fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878\": container with ID starting with fe09190724d2961e5403246d4cb05d0b89f5b3a73160338510fda88bab646878 not found: ID does not exist"
Jul 16 17:50:16.876966 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.876943    2563 scope.go:117] "RemoveContainer" containerID="f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57"
Jul 16 17:50:16.877149 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:50:16.877130    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57\": container with ID starting with f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57 not found: ID does not exist" containerID="f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57"
Jul 16 17:50:16.877192 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.877155    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57"} err="failed to get container status \"f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57\": rpc error: code = NotFound desc = could not find container \"f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57\": container with ID starting with f14caea1329b77d91f01eb98d9fffd6837de13412f8f74c68088eca9548adf57 not found: ID does not exist"
Jul 16 17:50:16.881450 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.881434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/580d18b7-efd9-410c-acda-0e237b5941d8-proxy-tls\") pod \"580d18b7-efd9-410c-acda-0e237b5941d8\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") "
Jul 16 17:50:16.881512 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.881464    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-497zk\" (UniqueName: \"kubernetes.io/projected/580d18b7-efd9-410c-acda-0e237b5941d8-kube-api-access-497zk\") pod \"580d18b7-efd9-410c-acda-0e237b5941d8\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") "
Jul 16 17:50:16.881512 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.881491    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/580d18b7-efd9-410c-acda-0e237b5941d8-kserve-provision-location\") pod \"580d18b7-efd9-410c-acda-0e237b5941d8\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") "
Jul 16 17:50:16.881625 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.881516    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-cabundle-cert\") pod \"580d18b7-efd9-410c-acda-0e237b5941d8\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") "
Jul 16 17:50:16.881625 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.881576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-kube-rbac-proxy-sar-config\") pod \"580d18b7-efd9-410c-acda-0e237b5941d8\" (UID: \"580d18b7-efd9-410c-acda-0e237b5941d8\") "
Jul 16 17:50:16.881843 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.881817    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580d18b7-efd9-410c-acda-0e237b5941d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "580d18b7-efd9-410c-acda-0e237b5941d8" (UID: "580d18b7-efd9-410c-acda-0e237b5941d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:16.881955 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.881930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "580d18b7-efd9-410c-acda-0e237b5941d8" (UID: "580d18b7-efd9-410c-acda-0e237b5941d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:50:16.882070 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.882051    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "580d18b7-efd9-410c-acda-0e237b5941d8" (UID: "580d18b7-efd9-410c-acda-0e237b5941d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:50:16.883525 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.883502    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580d18b7-efd9-410c-acda-0e237b5941d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "580d18b7-efd9-410c-acda-0e237b5941d8" (UID: "580d18b7-efd9-410c-acda-0e237b5941d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:50:16.883627 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.883545    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580d18b7-efd9-410c-acda-0e237b5941d8-kube-api-access-497zk" (OuterVolumeSpecName: "kube-api-access-497zk") pod "580d18b7-efd9-410c-acda-0e237b5941d8" (UID: "580d18b7-efd9-410c-acda-0e237b5941d8"). InnerVolumeSpecName "kube-api-access-497zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:50:16.982235 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.982196    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:50:16.982235 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.982224    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/580d18b7-efd9-410c-acda-0e237b5941d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:50:16.982235 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.982236    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/580d18b7-efd9-410c-acda-0e237b5941d8-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:50:16.982235 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.982245    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-497zk\" (UniqueName: \"kubernetes.io/projected/580d18b7-efd9-410c-acda-0e237b5941d8-kube-api-access-497zk\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:50:16.982235 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:16.982253    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/580d18b7-efd9-410c-acda-0e237b5941d8-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:50:17.181235 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:17.181190    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"]
Jul 16 17:50:17.187110 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:17.187082    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-57df2-predictor-6bc9ffd7d4-bxwtb"]
Jul 16 17:50:17.673741 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:17.673700    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" path="/var/lib/kubelet/pods/580d18b7-efd9-410c-acda-0e237b5941d8/volumes"
Jul 16 17:50:22.862462 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:22.862427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:50:31.949953 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.949915    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"]
Jul 16 17:50:31.950363 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.950257    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container"
Jul 16 17:50:31.950363 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.950269    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container"
Jul 16 17:50:31.950363 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.950281    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="storage-initializer"
Jul 16 17:50:31.950363 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.950287    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="storage-initializer"
Jul 16 17:50:31.950363 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.950305    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kube-rbac-proxy"
Jul 16 17:50:31.950363 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.950311    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kube-rbac-proxy"
Jul 16 17:50:31.950578 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.950378    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kserve-container"
Jul 16 17:50:31.950578 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.950389    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="580d18b7-efd9-410c-acda-0e237b5941d8" containerName="kube-rbac-proxy"
Jul 16 17:50:31.955352 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.955328    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:31.958227 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.958202    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:50:31.958368 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.958338    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-c3a17-kube-rbac-proxy-sar-config\""
Jul 16 17:50:31.958368 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.958360    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 17:50:31.958492 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.958359    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-c3a17-predictor-serving-cert\""
Jul 16 17:50:31.966342 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:31.966318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"]
Jul 16 17:50:32.108788 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.108737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.108788 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.108790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbws\" (UniqueName: \"kubernetes.io/projected/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-api-access-gmbws\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.109006 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.108826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kserve-provision-location\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.109006 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.108904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc7b553-617d-4abe-9b21-aaa8054c93e5-proxy-tls\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.109006 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.108936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-cabundle-cert\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.210288 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.210181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.210288 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.210230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbws\" (UniqueName: \"kubernetes.io/projected/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-api-access-gmbws\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.210288 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.210259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kserve-provision-location\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.210601 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.210298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc7b553-617d-4abe-9b21-aaa8054c93e5-proxy-tls\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.210601 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.210321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-cabundle-cert\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.210794 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.210766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kserve-provision-location\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.211015 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.210984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.211117 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.210991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-cabundle-cert\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.212826 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.212808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc7b553-617d-4abe-9b21-aaa8054c93e5-proxy-tls\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.222999 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.222968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbws\" (UniqueName: \"kubernetes.io/projected/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-api-access-gmbws\") pod \"isvc-logger-raw-c3a17-predictor-d4997994d-8v68s\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.265966 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.265916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:32.406506 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.406481    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"]
Jul 16 17:50:32.408754 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:50:32.408724    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc7b553_617d_4abe_9b21_aaa8054c93e5.slice/crio-7e0af1b70baaaa10f56a1316508d7afec45d28b96f0c6734a6c9c79cd15be8c5 WatchSource:0}: Error finding container 7e0af1b70baaaa10f56a1316508d7afec45d28b96f0c6734a6c9c79cd15be8c5: Status 404 returned error can't find the container with id 7e0af1b70baaaa10f56a1316508d7afec45d28b96f0c6734a6c9c79cd15be8c5
Jul 16 17:50:32.923977 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.923936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" event={"ID":"4fc7b553-617d-4abe-9b21-aaa8054c93e5","Type":"ContainerStarted","Data":"680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487"}
Jul 16 17:50:32.924124 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:32.924021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" event={"ID":"4fc7b553-617d-4abe-9b21-aaa8054c93e5","Type":"ContainerStarted","Data":"7e0af1b70baaaa10f56a1316508d7afec45d28b96f0c6734a6c9c79cd15be8c5"}
Jul 16 17:50:36.944103 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:36.944070    2563 generic.go:356] "Generic (PLEG): container finished" podID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerID="680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487" exitCode=0
Jul 16 17:50:36.944493 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:36.944144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" event={"ID":"4fc7b553-617d-4abe-9b21-aaa8054c93e5","Type":"ContainerDied","Data":"680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487"}
Jul 16 17:50:37.953828 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:37.953794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" event={"ID":"4fc7b553-617d-4abe-9b21-aaa8054c93e5","Type":"ContainerStarted","Data":"698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845"}
Jul 16 17:50:37.954202 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:37.953860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" event={"ID":"4fc7b553-617d-4abe-9b21-aaa8054c93e5","Type":"ContainerStarted","Data":"4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9"}
Jul 16 17:50:37.954202 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:37.953883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" event={"ID":"4fc7b553-617d-4abe-9b21-aaa8054c93e5","Type":"ContainerStarted","Data":"6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f"}
Jul 16 17:50:37.954202 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:37.954181    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:37.954342 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:37.954245    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:37.954342 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:37.954280    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:37.955782 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:37.955744    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:50:37.956533 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:37.956494    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:37.976276 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:37.976211    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podStartSLOduration=3.110613284 podStartE2EDuration="6.976186946s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.865573662s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:50:37.974223964 +0000 UTC m=+1014.960759572" watchObservedRunningTime="2026-07-16 17:50:37.976186946 +0000 UTC m=+1014.962722553"
Jul 16 17:50:38.959385 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:38.959336    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:50:38.959904 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:38.959875    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:43.964009 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:43.963928    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:50:43.964709 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:43.964655    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:50:43.964921 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:43.964898    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:53.964862 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:53.964811    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:50:53.965322 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:50:53.965295    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:03.965009 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:03.964960    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:51:03.965418 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:03.965383    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:13.965452 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:13.965399    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:51:13.965907 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:13.965819    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:23.965442 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:23.965392    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:51:23.965978 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:23.965875    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:33.965419 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:33.965368    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:51:33.965912 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:33.965883    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:43.965442 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:43.965394    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:51:43.965943 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:43.965795    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:53.965241 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:53.965211    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:51:53.965661 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:51:53.965633    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:52:06.931739 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:06.931709    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c_15bc05f5-8881-495b-8510-ac0076c95f6f/kserve-container/0.log"
Jul 16 17:52:07.201964 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.201874    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"]
Jul 16 17:52:07.202268 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.202243    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" containerID="cri-o://6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f" gracePeriod=30
Jul 16 17:52:07.202375 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.202321    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" containerID="cri-o://698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845" gracePeriod=30
Jul 16 17:52:07.202438 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.202324    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kube-rbac-proxy" containerID="cri-o://4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9" gracePeriod=30
Jul 16 17:52:07.289201 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.289165    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"]
Jul 16 17:52:07.292882 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.292863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.296160 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.296131    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-b0db6-kube-rbac-proxy-sar-config\""
Jul 16 17:52:07.296291 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.296131    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-b0db6-predictor-serving-cert\""
Jul 16 17:52:07.306333 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.306302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"]
Jul 16 17:52:07.318450 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.318406    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"]
Jul 16 17:52:07.318748 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.318701    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c" podUID="15bc05f5-8881-495b-8510-ac0076c95f6f" containerName="kserve-container" containerID="cri-o://26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de" gracePeriod=30
Jul 16 17:52:07.318893 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.318817    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c" podUID="15bc05f5-8881-495b-8510-ac0076c95f6f" containerName="kube-rbac-proxy" containerID="cri-o://cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993" gracePeriod=30
Jul 16 17:52:07.340293 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.340191    2563 generic.go:356] "Generic (PLEG): container finished" podID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerID="4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9" exitCode=2
Jul 16 17:52:07.340293 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.340243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" event={"ID":"4fc7b553-617d-4abe-9b21-aaa8054c93e5","Type":"ContainerDied","Data":"4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9"}
Jul 16 17:52:07.399853 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.399819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.399853 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.399866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-proxy-tls\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.400122 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.399902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5zd\" (UniqueName: \"kubernetes.io/projected/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-api-access-vg5zd\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.400122 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.399949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.400122 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.399978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.501141 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.501101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-proxy-tls\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.501331 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.501165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5zd\" (UniqueName: \"kubernetes.io/projected/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-api-access-vg5zd\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.501331 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.501208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.501331 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.501256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.501331 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.501311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.502467 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.502414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.502821 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.502799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.502926 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.502908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.504653 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.504578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-proxy-tls\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.512123 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.512096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5zd\" (UniqueName: \"kubernetes.io/projected/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-api-access-vg5zd\") pod \"isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.564265 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.564235    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:52:07.602211 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.602178    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-rbac-proxy-sar-config\") pod \"15bc05f5-8881-495b-8510-ac0076c95f6f\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") "
Jul 16 17:52:07.602416 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.602247    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15bc05f5-8881-495b-8510-ac0076c95f6f-proxy-tls\") pod \"15bc05f5-8881-495b-8510-ac0076c95f6f\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") "
Jul 16 17:52:07.602416 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.602320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwq7x\" (UniqueName: \"kubernetes.io/projected/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-api-access-lwq7x\") pod \"15bc05f5-8881-495b-8510-ac0076c95f6f\" (UID: \"15bc05f5-8881-495b-8510-ac0076c95f6f\") "
Jul 16 17:52:07.602668 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.602596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15bc05f5-8881-495b-8510-ac0076c95f6f" (UID: "15bc05f5-8881-495b-8510-ac0076c95f6f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:52:07.602668 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.602642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:07.604485 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.604457    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15bc05f5-8881-495b-8510-ac0076c95f6f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15bc05f5-8881-495b-8510-ac0076c95f6f" (UID: "15bc05f5-8881-495b-8510-ac0076c95f6f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:52:07.604764 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.604742    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-api-access-lwq7x" (OuterVolumeSpecName: "kube-api-access-lwq7x") pod "15bc05f5-8881-495b-8510-ac0076c95f6f" (UID: "15bc05f5-8881-495b-8510-ac0076c95f6f"). InnerVolumeSpecName "kube-api-access-lwq7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:52:07.704022 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.703793    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15bc05f5-8881-495b-8510-ac0076c95f6f-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:52:07.704022 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.703980    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwq7x\" (UniqueName: \"kubernetes.io/projected/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-api-access-lwq7x\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:52:07.704022 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.704006    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15bc05f5-8881-495b-8510-ac0076c95f6f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:52:07.749317 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:07.749288    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"]
Jul 16 17:52:07.750829 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:52:07.750792    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1049ca5f_1d2d_4f8e_adc8_1ecf32019676.slice/crio-9ed50b5e6dcab0da9fcdf43005745c19109903ec0abfd60a6a2af0bb1fd6d76d WatchSource:0}: Error finding container 9ed50b5e6dcab0da9fcdf43005745c19109903ec0abfd60a6a2af0bb1fd6d76d: Status 404 returned error can't find the container with id 9ed50b5e6dcab0da9fcdf43005745c19109903ec0abfd60a6a2af0bb1fd6d76d
Jul 16 17:52:08.346753 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.346709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" event={"ID":"1049ca5f-1d2d-4f8e-adc8-1ecf32019676","Type":"ContainerStarted","Data":"cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c"}
Jul 16 17:52:08.347205 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.346792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" event={"ID":"1049ca5f-1d2d-4f8e-adc8-1ecf32019676","Type":"ContainerStarted","Data":"9ed50b5e6dcab0da9fcdf43005745c19109903ec0abfd60a6a2af0bb1fd6d76d"}
Jul 16 17:52:08.348365 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.348330    2563 generic.go:356] "Generic (PLEG): container finished" podID="15bc05f5-8881-495b-8510-ac0076c95f6f" containerID="cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993" exitCode=2
Jul 16 17:52:08.348365 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.348365    2563 generic.go:356] "Generic (PLEG): container finished" podID="15bc05f5-8881-495b-8510-ac0076c95f6f" containerID="26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de" exitCode=2
Jul 16 17:52:08.348540 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.348371    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"
Jul 16 17:52:08.348540 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.348403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c" event={"ID":"15bc05f5-8881-495b-8510-ac0076c95f6f","Type":"ContainerDied","Data":"cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993"}
Jul 16 17:52:08.348540 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.348466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c" event={"ID":"15bc05f5-8881-495b-8510-ac0076c95f6f","Type":"ContainerDied","Data":"26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de"}
Jul 16 17:52:08.348540 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.348507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c" event={"ID":"15bc05f5-8881-495b-8510-ac0076c95f6f","Type":"ContainerDied","Data":"e976729600962b8910f24eb044149603b9077a1950c2133fee537c8097782e0b"}
Jul 16 17:52:08.348742 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.348563    2563 scope.go:117] "RemoveContainer" containerID="cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993"
Jul 16 17:52:08.357415 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.357393    2563 scope.go:117] "RemoveContainer" containerID="26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de"
Jul 16 17:52:08.365756 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.365732    2563 scope.go:117] "RemoveContainer" containerID="cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993"
Jul 16 17:52:08.366100 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:52:08.366079    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993\": container with ID starting with cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993 not found: ID does not exist" containerID="cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993"
Jul 16 17:52:08.366174 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.366110    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993"} err="failed to get container status \"cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993\": rpc error: code = NotFound desc = could not find container \"cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993\": container with ID starting with cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993 not found: ID does not exist"
Jul 16 17:52:08.366174 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.366134    2563 scope.go:117] "RemoveContainer" containerID="26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de"
Jul 16 17:52:08.366424 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:52:08.366403    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de\": container with ID starting with 26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de not found: ID does not exist" containerID="26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de"
Jul 16 17:52:08.366488 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.366435    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de"} err="failed to get container status \"26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de\": rpc error: code = NotFound desc = could not find container \"26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de\": container with ID starting with 26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de not found: ID does not exist"
Jul 16 17:52:08.366488 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.366461    2563 scope.go:117] "RemoveContainer" containerID="cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993"
Jul 16 17:52:08.366910 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.366884    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993"} err="failed to get container status \"cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993\": rpc error: code = NotFound desc = could not find container \"cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993\": container with ID starting with cbb59ef1ff8e344ef0ceaea9786e37fd4648442f081a4fe4be14d4b1a44bb993 not found: ID does not exist"
Jul 16 17:52:08.366987 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.366912    2563 scope.go:117] "RemoveContainer" containerID="26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de"
Jul 16 17:52:08.367180 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.367159    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de"} err="failed to get container status \"26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de\": rpc error: code = NotFound desc = could not find container \"26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de\": container with ID starting with 26e16236196f30cf6fedfd8a2d8579927bffc7a451c2f5e35fc444f198aef1de not found: ID does not exist"
Jul 16 17:52:08.412417 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.412373    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"]
Jul 16 17:52:08.423183 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.423147    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c3a17-predictor-6788446ffc-h9n8c"]
Jul 16 17:52:08.960401 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:08.960359    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 16 17:52:09.673243 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:09.673204    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15bc05f5-8881-495b-8510-ac0076c95f6f" path="/var/lib/kubelet/pods/15bc05f5-8881-495b-8510-ac0076c95f6f/volumes"
Jul 16 17:52:12.374865 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:12.374826    2563 generic.go:356] "Generic (PLEG): container finished" podID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerID="6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f" exitCode=0
Jul 16 17:52:12.375249 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:12.374897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" event={"ID":"4fc7b553-617d-4abe-9b21-aaa8054c93e5","Type":"ContainerDied","Data":"6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f"}
Jul 16 17:52:12.376431 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:12.376411    2563 generic.go:356] "Generic (PLEG): container finished" podID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerID="cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c" exitCode=0
Jul 16 17:52:12.376526 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:12.376446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" event={"ID":"1049ca5f-1d2d-4f8e-adc8-1ecf32019676","Type":"ContainerDied","Data":"cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c"}
Jul 16 17:52:13.383463 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:13.383429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" event={"ID":"1049ca5f-1d2d-4f8e-adc8-1ecf32019676","Type":"ContainerStarted","Data":"807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e"}
Jul 16 17:52:13.383872 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:13.383491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" event={"ID":"1049ca5f-1d2d-4f8e-adc8-1ecf32019676","Type":"ContainerStarted","Data":"9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2"}
Jul 16 17:52:13.383872 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:13.383785    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:13.383967 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:13.383946    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:13.385426 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:13.385394    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:52:13.406836 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:13.406784    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podStartSLOduration=2.7496051870000002 podStartE2EDuration="6.406766071s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.657160884s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:52:13.405028083 +0000 UTC m=+1110.391563690" watchObservedRunningTime="2026-07-16 17:52:13.406766071 +0000 UTC m=+1110.393301674"
Jul 16 17:52:13.960318 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:13.960276    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 16 17:52:13.965639 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:13.965601    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:52:13.965794 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:13.965654    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:52:14.388763 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:14.388718    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:52:18.959802 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:18.959760    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 16 17:52:18.960206 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:18.959895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:52:19.395205 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:19.395173    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:52:19.395732 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:19.395704    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:52:23.959528 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:23.959488    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 16 17:52:23.965251 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:23.965213    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:52:23.966438 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:23.966415    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:52:28.959830 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:28.959782    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 16 17:52:29.395923 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:29.395882    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:52:33.960072 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:33.960029    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 16 17:52:33.965257 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:33.965225    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:52:33.965390 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:33.965362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:52:33.965730 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:33.965707    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 16 17:52:33.965837 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:33.965824    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:52:37.366159 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.366131    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:52:37.466948 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.466853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-rbac-proxy-sar-config\") pod \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") "
Jul 16 17:52:37.466948 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.466918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kserve-provision-location\") pod \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") "
Jul 16 17:52:37.467182 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.466952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc7b553-617d-4abe-9b21-aaa8054c93e5-proxy-tls\") pod \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") "
Jul 16 17:52:37.467182 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.467058    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmbws\" (UniqueName: \"kubernetes.io/projected/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-api-access-gmbws\") pod \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") "
Jul 16 17:52:37.467182 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.467107    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-cabundle-cert\") pod \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\" (UID: \"4fc7b553-617d-4abe-9b21-aaa8054c93e5\") "
Jul 16 17:52:37.467335 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.467270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4fc7b553-617d-4abe-9b21-aaa8054c93e5" (UID: "4fc7b553-617d-4abe-9b21-aaa8054c93e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:52:37.467335 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.467286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4fc7b553-617d-4abe-9b21-aaa8054c93e5" (UID: "4fc7b553-617d-4abe-9b21-aaa8054c93e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:52:37.467613 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.467594    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:52:37.467672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.467621    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:52:37.467672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.467583    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4fc7b553-617d-4abe-9b21-aaa8054c93e5" (UID: "4fc7b553-617d-4abe-9b21-aaa8054c93e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:52:37.469173 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.469147    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc7b553-617d-4abe-9b21-aaa8054c93e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4fc7b553-617d-4abe-9b21-aaa8054c93e5" (UID: "4fc7b553-617d-4abe-9b21-aaa8054c93e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:52:37.469283 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.469260    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-api-access-gmbws" (OuterVolumeSpecName: "kube-api-access-gmbws") pod "4fc7b553-617d-4abe-9b21-aaa8054c93e5" (UID: "4fc7b553-617d-4abe-9b21-aaa8054c93e5"). InnerVolumeSpecName "kube-api-access-gmbws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:52:37.507388 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.507355    2563 generic.go:356] "Generic (PLEG): container finished" podID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerID="698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845" exitCode=0
Jul 16 17:52:37.507597 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.507436    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"
Jul 16 17:52:37.507597 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.507441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" event={"ID":"4fc7b553-617d-4abe-9b21-aaa8054c93e5","Type":"ContainerDied","Data":"698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845"}
Jul 16 17:52:37.507597 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.507578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s" event={"ID":"4fc7b553-617d-4abe-9b21-aaa8054c93e5","Type":"ContainerDied","Data":"7e0af1b70baaaa10f56a1316508d7afec45d28b96f0c6734a6c9c79cd15be8c5"}
Jul 16 17:52:37.507749 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.507626    2563 scope.go:117] "RemoveContainer" containerID="698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845"
Jul 16 17:52:37.517045 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.517015    2563 scope.go:117] "RemoveContainer" containerID="4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9"
Jul 16 17:52:37.525070 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.525045    2563 scope.go:117] "RemoveContainer" containerID="6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f"
Jul 16 17:52:37.533185 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.533156    2563 scope.go:117] "RemoveContainer" containerID="680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487"
Jul 16 17:52:37.533729 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.533708    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"]
Jul 16 17:52:37.538172 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.538150    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c3a17-predictor-d4997994d-8v68s"]
Jul 16 17:52:37.541524 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.541510    2563 scope.go:117] "RemoveContainer" containerID="698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845"
Jul 16 17:52:37.544366 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:52:37.544335    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845\": container with ID starting with 698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845 not found: ID does not exist" containerID="698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845"
Jul 16 17:52:37.544490 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.544373    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845"} err="failed to get container status \"698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845\": rpc error: code = NotFound desc = could not find container \"698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845\": container with ID starting with 698dfc9f3ba66f66f2ce7fcb3fbd0181a691c908f5625dd0dd306a8f8efbe845 not found: ID does not exist"
Jul 16 17:52:37.544490 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.544400    2563 scope.go:117] "RemoveContainer" containerID="4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9"
Jul 16 17:52:37.544983 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:52:37.544960    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9\": container with ID starting with 4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9 not found: ID does not exist" containerID="4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9"
Jul 16 17:52:37.545036 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.544992    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9"} err="failed to get container status \"4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9\": rpc error: code = NotFound desc = could not find container \"4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9\": container with ID starting with 4578dbb0e6f68b97a1c5b1cd1a15f53b6712fb0fc84d946ade035f384ab0aba9 not found: ID does not exist"
Jul 16 17:52:37.545036 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.545012    2563 scope.go:117] "RemoveContainer" containerID="6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f"
Jul 16 17:52:37.545296 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:52:37.545281    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f\": container with ID starting with 6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f not found: ID does not exist" containerID="6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f"
Jul 16 17:52:37.545338 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.545299    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f"} err="failed to get container status \"6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f\": rpc error: code = NotFound desc = could not find container \"6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f\": container with ID starting with 6bfd89af3d27935f98c006e4d59c286a1e3f220311eab8a5c8814b90ec22f34f not found: ID does not exist"
Jul 16 17:52:37.545338 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.545312    2563 scope.go:117] "RemoveContainer" containerID="680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487"
Jul 16 17:52:37.545508 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:52:37.545492    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487\": container with ID starting with 680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487 not found: ID does not exist" containerID="680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487"
Jul 16 17:52:37.545575 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.545510    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487"} err="failed to get container status \"680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487\": rpc error: code = NotFound desc = could not find container \"680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487\": container with ID starting with 680b7bd05e43091c3f9ac5e19f99f8f3e92c944fc3f46cdd90c298797d0bc487 not found: ID does not exist"
Jul 16 17:52:37.569000 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.568961    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc7b553-617d-4abe-9b21-aaa8054c93e5-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:52:37.569000 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.568997    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmbws\" (UniqueName: \"kubernetes.io/projected/4fc7b553-617d-4abe-9b21-aaa8054c93e5-kube-api-access-gmbws\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:52:37.569123 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.569007    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc7b553-617d-4abe-9b21-aaa8054c93e5-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:52:37.673685 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:37.673654    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" path="/var/lib/kubelet/pods/4fc7b553-617d-4abe-9b21-aaa8054c93e5/volumes"
Jul 16 17:52:39.396479 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:39.396436    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:52:49.395835 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:49.395788    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:52:59.396136 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:52:59.396099    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:53:09.396093 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:09.396008    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:53:19.396246 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:19.396201    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:53:29.396304 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:29.396263    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:53:34.668871 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:34.668830    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:53:43.568928 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:43.568897    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:53:43.571721 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:43.571691    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:53:43.573146 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:43.573121    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:53:43.575702 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:43.575681    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:53:43.575858 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:43.575843    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:53:43.579744 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:43.579724    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:53:44.669834 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:44.669783    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:53:54.668777 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:53:54.668736    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:54:04.669137 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:04.669082    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:54:14.669685 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:14.669637    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:54:24.669205 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:24.669159    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:54:34.668888 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:34.668838    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:54:44.668778 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:44.668738    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:54:54.669728 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:54.669694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:54:57.448120 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:57.448086    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"]
Jul 16 17:54:57.448624 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:57.448371    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" containerID="cri-o://9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2" gracePeriod=30
Jul 16 17:54:57.448624 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:57.448411    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kube-rbac-proxy" containerID="cri-o://807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e" gracePeriod=30
Jul 16 17:54:58.082231 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:58.082195    2563 generic.go:356] "Generic (PLEG): container finished" podID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerID="807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e" exitCode=2
Jul 16 17:54:58.082400 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:58.082267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" event={"ID":"1049ca5f-1d2d-4f8e-adc8-1ecf32019676","Type":"ContainerDied","Data":"807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e"}
Jul 16 17:54:59.388910 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:54:59.388869    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 16 17:55:02.740524 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740484    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"]
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740817    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="storage-initializer"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740829    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="storage-initializer"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740844    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740851    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740858    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kube-rbac-proxy"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740864    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kube-rbac-proxy"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740871    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15bc05f5-8881-495b-8510-ac0076c95f6f" containerName="kserve-container"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740877    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15bc05f5-8881-495b-8510-ac0076c95f6f" containerName="kserve-container"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740887    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740892    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740901    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15bc05f5-8881-495b-8510-ac0076c95f6f" containerName="kube-rbac-proxy"
Jul 16 17:55:02.740903 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740906    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15bc05f5-8881-495b-8510-ac0076c95f6f" containerName="kube-rbac-proxy"
Jul 16 17:55:02.741257 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740958    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="15bc05f5-8881-495b-8510-ac0076c95f6f" containerName="kube-rbac-proxy"
Jul 16 17:55:02.741257 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740966    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="agent"
Jul 16 17:55:02.741257 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740973    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kserve-container"
Jul 16 17:55:02.741257 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740979    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="15bc05f5-8881-495b-8510-ac0076c95f6f" containerName="kserve-container"
Jul 16 17:55:02.741257 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.740986    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc7b553-617d-4abe-9b21-aaa8054c93e5" containerName="kube-rbac-proxy"
Jul 16 17:55:02.744113 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.744097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.747150 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.747123    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-aae39d-predictor-serving-cert\""
Jul 16 17:55:02.747274 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.747134    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-aae39d-kube-rbac-proxy-sar-config\""
Jul 16 17:55:02.756697 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.756673    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"]
Jul 16 17:55:02.845111 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.845078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-cabundle-cert\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.845292 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.845120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f835a159-e668-421c-b478-2a7d9761aa01-kserve-provision-location\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.845292 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.845146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-kube-rbac-proxy-sar-config\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.845292 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.845179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87sh4\" (UniqueName: \"kubernetes.io/projected/f835a159-e668-421c-b478-2a7d9761aa01-kube-api-access-87sh4\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.845292 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.845228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f835a159-e668-421c-b478-2a7d9761aa01-proxy-tls\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.945828 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.945780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f835a159-e668-421c-b478-2a7d9761aa01-kserve-provision-location\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.945828 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.945825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-kube-rbac-proxy-sar-config\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.946072 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.945852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87sh4\" (UniqueName: \"kubernetes.io/projected/f835a159-e668-421c-b478-2a7d9761aa01-kube-api-access-87sh4\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.946072 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.945897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f835a159-e668-421c-b478-2a7d9761aa01-proxy-tls\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.946072 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.945932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-cabundle-cert\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.946359 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.946333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f835a159-e668-421c-b478-2a7d9761aa01-kserve-provision-location\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.946607 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.946589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-kube-rbac-proxy-sar-config\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.946674 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.946658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-cabundle-cert\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.948337 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.948319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f835a159-e668-421c-b478-2a7d9761aa01-proxy-tls\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:02.957865 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:02.957836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87sh4\" (UniqueName: \"kubernetes.io/projected/f835a159-e668-421c-b478-2a7d9761aa01-kube-api-access-87sh4\") pod \"isvc-primary-aae39d-predictor-d67696c6f-46tks\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") " pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:03.053772 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:03.053682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:03.199075 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:03.199042    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"]
Jul 16 17:55:03.203479 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:55:03.203448    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf835a159_e668_421c_b478_2a7d9761aa01.slice/crio-92f46fc41b7478685059d2ef5e032b6ea4b4486964fddda4dfd9f8c795a1bad5 WatchSource:0}: Error finding container 92f46fc41b7478685059d2ef5e032b6ea4b4486964fddda4dfd9f8c795a1bad5: Status 404 returned error can't find the container with id 92f46fc41b7478685059d2ef5e032b6ea4b4486964fddda4dfd9f8c795a1bad5
Jul 16 17:55:04.106737 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:04.106702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" event={"ID":"f835a159-e668-421c-b478-2a7d9761aa01","Type":"ContainerStarted","Data":"28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647"}
Jul 16 17:55:04.107123 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:04.106764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" event={"ID":"f835a159-e668-421c-b478-2a7d9761aa01","Type":"ContainerStarted","Data":"92f46fc41b7478685059d2ef5e032b6ea4b4486964fddda4dfd9f8c795a1bad5"}
Jul 16 17:55:04.389187 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:04.389092    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 16 17:55:04.669410 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:04.669322    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 16 17:55:07.498622 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.498596    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:55:07.689264 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.689179    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg5zd\" (UniqueName: \"kubernetes.io/projected/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-api-access-vg5zd\") pod \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") "
Jul 16 17:55:07.689264 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.689257    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-cabundle-cert\") pod \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") "
Jul 16 17:55:07.689484 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.689415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-proxy-tls\") pod \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") "
Jul 16 17:55:07.689484 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.689468    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kserve-provision-location\") pod \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") "
Jul 16 17:55:07.689484 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.689485    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-rbac-proxy-sar-config\") pod \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\" (UID: \"1049ca5f-1d2d-4f8e-adc8-1ecf32019676\") "
Jul 16 17:55:07.689667 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.689584    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1049ca5f-1d2d-4f8e-adc8-1ecf32019676" (UID: "1049ca5f-1d2d-4f8e-adc8-1ecf32019676"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:55:07.689780 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.689759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1049ca5f-1d2d-4f8e-adc8-1ecf32019676" (UID: "1049ca5f-1d2d-4f8e-adc8-1ecf32019676"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:55:07.689871 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.689852    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:55:07.689914 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.689884    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:55:07.689914 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.689854    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1049ca5f-1d2d-4f8e-adc8-1ecf32019676" (UID: "1049ca5f-1d2d-4f8e-adc8-1ecf32019676"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:55:07.691322 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.691298    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-api-access-vg5zd" (OuterVolumeSpecName: "kube-api-access-vg5zd") pod "1049ca5f-1d2d-4f8e-adc8-1ecf32019676" (UID: "1049ca5f-1d2d-4f8e-adc8-1ecf32019676"). InnerVolumeSpecName "kube-api-access-vg5zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:55:07.691397 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.691369    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1049ca5f-1d2d-4f8e-adc8-1ecf32019676" (UID: "1049ca5f-1d2d-4f8e-adc8-1ecf32019676"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:55:07.790591 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.790528    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:55:07.790591 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.790587    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:55:07.790591 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:07.790598    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5zd\" (UniqueName: \"kubernetes.io/projected/1049ca5f-1d2d-4f8e-adc8-1ecf32019676-kube-api-access-vg5zd\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:55:08.125805 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.125769    2563 generic.go:356] "Generic (PLEG): container finished" podID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerID="9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2" exitCode=0
Jul 16 17:55:08.125989 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.125830    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"
Jul 16 17:55:08.125989 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.125857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" event={"ID":"1049ca5f-1d2d-4f8e-adc8-1ecf32019676","Type":"ContainerDied","Data":"9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2"}
Jul 16 17:55:08.125989 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.125935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6" event={"ID":"1049ca5f-1d2d-4f8e-adc8-1ecf32019676","Type":"ContainerDied","Data":"9ed50b5e6dcab0da9fcdf43005745c19109903ec0abfd60a6a2af0bb1fd6d76d"}
Jul 16 17:55:08.125989 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.125973    2563 scope.go:117] "RemoveContainer" containerID="807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e"
Jul 16 17:55:08.127428 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.127403    2563 generic.go:356] "Generic (PLEG): container finished" podID="f835a159-e668-421c-b478-2a7d9761aa01" containerID="28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647" exitCode=0
Jul 16 17:55:08.127528 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.127437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" event={"ID":"f835a159-e668-421c-b478-2a7d9761aa01","Type":"ContainerDied","Data":"28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647"}
Jul 16 17:55:08.135413 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.135390    2563 scope.go:117] "RemoveContainer" containerID="9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2"
Jul 16 17:55:08.143727 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.143711    2563 scope.go:117] "RemoveContainer" containerID="cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c"
Jul 16 17:55:08.152386 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.152365    2563 scope.go:117] "RemoveContainer" containerID="807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e"
Jul 16 17:55:08.152684 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:55:08.152665    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e\": container with ID starting with 807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e not found: ID does not exist" containerID="807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e"
Jul 16 17:55:08.152772 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.152697    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e"} err="failed to get container status \"807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e\": rpc error: code = NotFound desc = could not find container \"807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e\": container with ID starting with 807f4f1f7bcc43176048d18c92b4a623cbd1c8a17719fddf342a77a6eb46340e not found: ID does not exist"
Jul 16 17:55:08.152772 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.152721    2563 scope.go:117] "RemoveContainer" containerID="9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2"
Jul 16 17:55:08.153028 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:55:08.153008    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2\": container with ID starting with 9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2 not found: ID does not exist" containerID="9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2"
Jul 16 17:55:08.153116 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.153036    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2"} err="failed to get container status \"9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2\": rpc error: code = NotFound desc = could not find container \"9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2\": container with ID starting with 9957095a728c1402ba07672a2c2fa10918e216c450b2dee7c19e1ed37959e2e2 not found: ID does not exist"
Jul 16 17:55:08.153116 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.153056    2563 scope.go:117] "RemoveContainer" containerID="cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c"
Jul 16 17:55:08.153342 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:55:08.153317    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c\": container with ID starting with cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c not found: ID does not exist" containerID="cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c"
Jul 16 17:55:08.153383 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.153353    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c"} err="failed to get container status \"cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c\": rpc error: code = NotFound desc = could not find container \"cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c\": container with ID starting with cc72f0e5024552bbeba02119232b79129ddeffaabf7a3228e71c731646ac109c not found: ID does not exist"
Jul 16 17:55:08.170197 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.170172    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"]
Jul 16 17:55:08.174981 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:08.174959    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b0db6-predictor-667dc794db-fp5w6"]
Jul 16 17:55:09.135062 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:09.135031    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" event={"ID":"f835a159-e668-421c-b478-2a7d9761aa01","Type":"ContainerStarted","Data":"632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557"}
Jul 16 17:55:09.135451 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:09.135087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" event={"ID":"f835a159-e668-421c-b478-2a7d9761aa01","Type":"ContainerStarted","Data":"14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5"}
Jul 16 17:55:09.135451 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:09.135396    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:09.135586 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:09.135569    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:09.136899 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:09.136876    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 16 17:55:09.162705 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:09.162651    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podStartSLOduration=3.2626983210000002 podStartE2EDuration="7.162635876s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.899937555s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:55:09.160522001 +0000 UTC m=+1286.147057610" watchObservedRunningTime="2026-07-16 17:55:09.162635876 +0000 UTC m=+1286.149171481"
Jul 16 17:55:09.673937 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:09.673894    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" path="/var/lib/kubelet/pods/1049ca5f-1d2d-4f8e-adc8-1ecf32019676/volumes"
Jul 16 17:55:10.139321 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:10.139282    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 16 17:55:15.145454 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:15.145427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:55:15.145863 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:15.145828    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 16 17:55:25.146827 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:25.146786    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 16 17:55:35.146342 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:35.146297    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 16 17:55:45.146137 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:45.146097    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 16 17:55:55.146672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:55:55.146627    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 16 17:56:05.145919 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:05.145806    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 16 17:56:15.146838 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:15.146785    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 16 17:56:25.147395 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:25.147359    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:56:33.046099 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.046062    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"]
Jul 16 17:56:33.046485 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.046410    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kube-rbac-proxy"
Jul 16 17:56:33.046485 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.046421    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kube-rbac-proxy"
Jul 16 17:56:33.046485 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.046440    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="storage-initializer"
Jul 16 17:56:33.046485 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.046446    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="storage-initializer"
Jul 16 17:56:33.046485 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.046462    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container"
Jul 16 17:56:33.046485 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.046467    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container"
Jul 16 17:56:33.046717 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.046518    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kube-rbac-proxy"
Jul 16 17:56:33.046717 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.046529    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1049ca5f-1d2d-4f8e-adc8-1ecf32019676" containerName="kserve-container"
Jul 16 17:56:33.049641 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.049623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.052442 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.052416    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-aae39d-dockercfg-jq9l9\""
Jul 16 17:56:33.052626 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.052466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-aae39d\""
Jul 16 17:56:33.053144 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.053123    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-aae39d-kube-rbac-proxy-sar-config\""
Jul 16 17:56:33.053234 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.053156    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-aae39d-predictor-serving-cert\""
Jul 16 17:56:33.062449 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.062419    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"]
Jul 16 17:56:33.063911 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.063885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.063999 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.063926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knw4w\" (UniqueName: \"kubernetes.io/projected/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-api-access-knw4w\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.063999 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.063961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f03f2216-13c3-453c-bad0-0b70c3f64cbe-proxy-tls\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.063999 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.063983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-cabundle-cert\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.064112 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.064031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kserve-provision-location\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.164987 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.164944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kserve-provision-location\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.165172 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.165022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.165172 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.165054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knw4w\" (UniqueName: \"kubernetes.io/projected/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-api-access-knw4w\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.165172 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.165083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f03f2216-13c3-453c-bad0-0b70c3f64cbe-proxy-tls\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.165172 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.165111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-cabundle-cert\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.165370 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:56:33.165202    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-serving-cert: secret "isvc-secondary-aae39d-predictor-serving-cert" not found
Jul 16 17:56:33.165370 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:56:33.165292    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03f2216-13c3-453c-bad0-0b70c3f64cbe-proxy-tls podName:f03f2216-13c3-453c-bad0-0b70c3f64cbe nodeName:}" failed. No retries permitted until 2026-07-16 17:56:33.665275216 +0000 UTC m=+1370.651810800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f03f2216-13c3-453c-bad0-0b70c3f64cbe-proxy-tls") pod "isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd" (UID: "f03f2216-13c3-453c-bad0-0b70c3f64cbe") : secret "isvc-secondary-aae39d-predictor-serving-cert" not found
Jul 16 17:56:33.165462 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.165389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kserve-provision-location\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.165845 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.165825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.165887 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.165825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-cabundle-cert\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.174213 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.174187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knw4w\" (UniqueName: \"kubernetes.io/projected/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-api-access-knw4w\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.668110 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.668068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f03f2216-13c3-453c-bad0-0b70c3f64cbe-proxy-tls\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.670793 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.670755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f03f2216-13c3-453c-bad0-0b70c3f64cbe-proxy-tls\") pod \"isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") " pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:33.961589 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:33.961481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:34.110662 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:34.110634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"]
Jul 16 17:56:34.111924 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:56:34.111892    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03f2216_13c3_453c_bad0_0b70c3f64cbe.slice/crio-488da8fb51a7e47d46327a0f9d896892b2ff29dedd0b7a4256e95a8ed781ba74 WatchSource:0}: Error finding container 488da8fb51a7e47d46327a0f9d896892b2ff29dedd0b7a4256e95a8ed781ba74: Status 404 returned error can't find the container with id 488da8fb51a7e47d46327a0f9d896892b2ff29dedd0b7a4256e95a8ed781ba74
Jul 16 17:56:34.481708 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:34.481671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd" event={"ID":"f03f2216-13c3-453c-bad0-0b70c3f64cbe","Type":"ContainerStarted","Data":"7d74c162cc17683daa741a0534f99beae44f7db2c649425c6c6cc114c488a834"}
Jul 16 17:56:34.481864 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:34.481732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd" event={"ID":"f03f2216-13c3-453c-bad0-0b70c3f64cbe","Type":"ContainerStarted","Data":"488da8fb51a7e47d46327a0f9d896892b2ff29dedd0b7a4256e95a8ed781ba74"}
Jul 16 17:56:40.509219 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:40.509191    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd_f03f2216-13c3-453c-bad0-0b70c3f64cbe/storage-initializer/0.log"
Jul 16 17:56:40.509667 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:40.509229    2563 generic.go:356] "Generic (PLEG): container finished" podID="f03f2216-13c3-453c-bad0-0b70c3f64cbe" containerID="7d74c162cc17683daa741a0534f99beae44f7db2c649425c6c6cc114c488a834" exitCode=1
Jul 16 17:56:40.509667 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:40.509268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd" event={"ID":"f03f2216-13c3-453c-bad0-0b70c3f64cbe","Type":"ContainerDied","Data":"7d74c162cc17683daa741a0534f99beae44f7db2c649425c6c6cc114c488a834"}
Jul 16 17:56:41.515508 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:41.515483    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd_f03f2216-13c3-453c-bad0-0b70c3f64cbe/storage-initializer/0.log"
Jul 16 17:56:41.515915 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:41.515591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd" event={"ID":"f03f2216-13c3-453c-bad0-0b70c3f64cbe","Type":"ContainerStarted","Data":"f194bc92b8d9d6865a21337b94a2c9cdb795008d3399815905fd015e58d05b46"}
Jul 16 17:56:46.537674 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:46.537643    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd_f03f2216-13c3-453c-bad0-0b70c3f64cbe/storage-initializer/1.log"
Jul 16 17:56:46.538080 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:46.538065    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd_f03f2216-13c3-453c-bad0-0b70c3f64cbe/storage-initializer/0.log"
Jul 16 17:56:46.538122 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:46.538098    2563 generic.go:356] "Generic (PLEG): container finished" podID="f03f2216-13c3-453c-bad0-0b70c3f64cbe" containerID="f194bc92b8d9d6865a21337b94a2c9cdb795008d3399815905fd015e58d05b46" exitCode=1
Jul 16 17:56:46.538179 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:46.538164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd" event={"ID":"f03f2216-13c3-453c-bad0-0b70c3f64cbe","Type":"ContainerDied","Data":"f194bc92b8d9d6865a21337b94a2c9cdb795008d3399815905fd015e58d05b46"}
Jul 16 17:56:46.538235 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:46.538224    2563 scope.go:117] "RemoveContainer" containerID="7d74c162cc17683daa741a0534f99beae44f7db2c649425c6c6cc114c488a834"
Jul 16 17:56:46.538636 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:46.538613    2563 scope.go:117] "RemoveContainer" containerID="7d74c162cc17683daa741a0534f99beae44f7db2c649425c6c6cc114c488a834"
Jul 16 17:56:46.549215 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:56:46.549183    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd_kserve-ci-e2e-test(f03f2216-13c3-453c-bad0-0b70c3f64cbe)\"" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd" podUID="f03f2216-13c3-453c-bad0-0b70c3f64cbe"
Jul 16 17:56:46.549330 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:56:46.549302    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd_kserve-ci-e2e-test_f03f2216-13c3-453c-bad0-0b70c3f64cbe_0 in pod sandbox 488da8fb51a7e47d46327a0f9d896892b2ff29dedd0b7a4256e95a8ed781ba74 from index: no such id: '7d74c162cc17683daa741a0534f99beae44f7db2c649425c6c6cc114c488a834'" containerID="7d74c162cc17683daa741a0534f99beae44f7db2c649425c6c6cc114c488a834"
Jul 16 17:56:46.549381 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:46.549345    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d74c162cc17683daa741a0534f99beae44f7db2c649425c6c6cc114c488a834"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd_kserve-ci-e2e-test_f03f2216-13c3-453c-bad0-0b70c3f64cbe_0 in pod sandbox 488da8fb51a7e47d46327a0f9d896892b2ff29dedd0b7a4256e95a8ed781ba74 from index: no such id: '7d74c162cc17683daa741a0534f99beae44f7db2c649425c6c6cc114c488a834'"
Jul 16 17:56:47.544168 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:47.544138    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd_f03f2216-13c3-453c-bad0-0b70c3f64cbe/storage-initializer/1.log"
Jul 16 17:56:51.017450 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.017410    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"]
Jul 16 17:56:51.017942 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.017735    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" containerID="cri-o://14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5" gracePeriod=30
Jul 16 17:56:51.017942 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.017807    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kube-rbac-proxy" containerID="cri-o://632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557" gracePeriod=30
Jul 16 17:56:51.098371 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.098336    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"]
Jul 16 17:56:51.238015 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.237989    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd_f03f2216-13c3-453c-bad0-0b70c3f64cbe/storage-initializer/1.log"
Jul 16 17:56:51.240229 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.240209    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:51.319033 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.318935    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-cabundle-cert\") pod \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") "
Jul 16 17:56:51.319033 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.318980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-rbac-proxy-sar-config\") pod \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") "
Jul 16 17:56:51.319234 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.319039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kserve-provision-location\") pod \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") "
Jul 16 17:56:51.319234 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.319063    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knw4w\" (UniqueName: \"kubernetes.io/projected/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-api-access-knw4w\") pod \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") "
Jul 16 17:56:51.319234 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.319102    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f03f2216-13c3-453c-bad0-0b70c3f64cbe-proxy-tls\") pod \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\" (UID: \"f03f2216-13c3-453c-bad0-0b70c3f64cbe\") "
Jul 16 17:56:51.319414 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.319384    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f03f2216-13c3-453c-bad0-0b70c3f64cbe" (UID: "f03f2216-13c3-453c-bad0-0b70c3f64cbe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:56:51.319468 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.319405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f03f2216-13c3-453c-bad0-0b70c3f64cbe" (UID: "f03f2216-13c3-453c-bad0-0b70c3f64cbe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:56:51.319468 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.319410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f03f2216-13c3-453c-bad0-0b70c3f64cbe" (UID: "f03f2216-13c3-453c-bad0-0b70c3f64cbe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:56:51.321404 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.321371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-api-access-knw4w" (OuterVolumeSpecName: "kube-api-access-knw4w") pod "f03f2216-13c3-453c-bad0-0b70c3f64cbe" (UID: "f03f2216-13c3-453c-bad0-0b70c3f64cbe"). InnerVolumeSpecName "kube-api-access-knw4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:56:51.321404 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.321373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03f2216-13c3-453c-bad0-0b70c3f64cbe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f03f2216-13c3-453c-bad0-0b70c3f64cbe" (UID: "f03f2216-13c3-453c-bad0-0b70c3f64cbe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:56:51.420333 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.420276    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:56:51.420333 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.420326    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:56:51.420333 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.420340    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:56:51.420333 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.420350    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knw4w\" (UniqueName: \"kubernetes.io/projected/f03f2216-13c3-453c-bad0-0b70c3f64cbe-kube-api-access-knw4w\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:56:51.420692 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.420359    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f03f2216-13c3-453c-bad0-0b70c3f64cbe-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:56:51.564542 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.564496    2563 generic.go:356] "Generic (PLEG): container finished" podID="f835a159-e668-421c-b478-2a7d9761aa01" containerID="632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557" exitCode=2
Jul 16 17:56:51.564746 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.564586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" event={"ID":"f835a159-e668-421c-b478-2a7d9761aa01","Type":"ContainerDied","Data":"632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557"}
Jul 16 17:56:51.565941 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.565920    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd_f03f2216-13c3-453c-bad0-0b70c3f64cbe/storage-initializer/1.log"
Jul 16 17:56:51.566070 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.565971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd" event={"ID":"f03f2216-13c3-453c-bad0-0b70c3f64cbe","Type":"ContainerDied","Data":"488da8fb51a7e47d46327a0f9d896892b2ff29dedd0b7a4256e95a8ed781ba74"}
Jul 16 17:56:51.566070 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.566014    2563 scope.go:117] "RemoveContainer" containerID="f194bc92b8d9d6865a21337b94a2c9cdb795008d3399815905fd015e58d05b46"
Jul 16 17:56:51.566070 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.566026    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"
Jul 16 17:56:51.645372 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.645320    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"]
Jul 16 17:56:51.657165 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.657129    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-aae39d-predictor-5d8d6d5c86-2llkd"]
Jul 16 17:56:51.673969 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:51.673939    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03f2216-13c3-453c-bad0-0b70c3f64cbe" path="/var/lib/kubelet/pods/f03f2216-13c3-453c-bad0-0b70c3f64cbe/volumes"
Jul 16 17:56:55.140111 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:55.140069    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 16 17:56:55.146449 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:55.146406    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 16 17:56:56.184050 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.184021    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:56:56.249600 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.249562    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"]
Jul 16 17:56:56.249888 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.249876    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kube-rbac-proxy"
Jul 16 17:56:56.249931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.249890    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kube-rbac-proxy"
Jul 16 17:56:56.249931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.249897    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03f2216-13c3-453c-bad0-0b70c3f64cbe" containerName="storage-initializer"
Jul 16 17:56:56.249931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.249904    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03f2216-13c3-453c-bad0-0b70c3f64cbe" containerName="storage-initializer"
Jul 16 17:56:56.249931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.249919    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container"
Jul 16 17:56:56.249931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.249925    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container"
Jul 16 17:56:56.250097 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.249937    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="storage-initializer"
Jul 16 17:56:56.250097 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.249942    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="storage-initializer"
Jul 16 17:56:56.250097 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.249991    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kube-rbac-proxy"
Jul 16 17:56:56.250097 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.250000    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03f2216-13c3-453c-bad0-0b70c3f64cbe" containerName="storage-initializer"
Jul 16 17:56:56.250097 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.250008    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f835a159-e668-421c-b478-2a7d9761aa01" containerName="kserve-container"
Jul 16 17:56:56.250097 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.250058    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03f2216-13c3-453c-bad0-0b70c3f64cbe" containerName="storage-initializer"
Jul 16 17:56:56.250097 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.250064    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03f2216-13c3-453c-bad0-0b70c3f64cbe" containerName="storage-initializer"
Jul 16 17:56:56.250289 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.250124    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03f2216-13c3-453c-bad0-0b70c3f64cbe" containerName="storage-initializer"
Jul 16 17:56:56.252885 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.252866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.256443 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.256375    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-22eec6-kube-rbac-proxy-sar-config\""
Jul 16 17:56:56.256618 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.256604    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-22eec6\""
Jul 16 17:56:56.256677 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.256656    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-22eec6-predictor-serving-cert\""
Jul 16 17:56:56.256732 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.256666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-22eec6-dockercfg-lfb9w\""
Jul 16 17:56:56.261762 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.261738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87sh4\" (UniqueName: \"kubernetes.io/projected/f835a159-e668-421c-b478-2a7d9761aa01-kube-api-access-87sh4\") pod \"f835a159-e668-421c-b478-2a7d9761aa01\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") "
Jul 16 17:56:56.261881 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.261804    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-cabundle-cert\") pod \"f835a159-e668-421c-b478-2a7d9761aa01\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") "
Jul 16 17:56:56.261881 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.261822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f835a159-e668-421c-b478-2a7d9761aa01-proxy-tls\") pod \"f835a159-e668-421c-b478-2a7d9761aa01\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") "
Jul 16 17:56:56.261881 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.261860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-kube-rbac-proxy-sar-config\") pod \"f835a159-e668-421c-b478-2a7d9761aa01\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") "
Jul 16 17:56:56.262072 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.261893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f835a159-e668-421c-b478-2a7d9761aa01-kserve-provision-location\") pod \"f835a159-e668-421c-b478-2a7d9761aa01\" (UID: \"f835a159-e668-421c-b478-2a7d9761aa01\") "
Jul 16 17:56:56.262217 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.262188    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f835a159-e668-421c-b478-2a7d9761aa01" (UID: "f835a159-e668-421c-b478-2a7d9761aa01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:56:56.262337 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.262239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f835a159-e668-421c-b478-2a7d9761aa01" (UID: "f835a159-e668-421c-b478-2a7d9761aa01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:56:56.262337 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.262251    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f835a159-e668-421c-b478-2a7d9761aa01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f835a159-e668-421c-b478-2a7d9761aa01" (UID: "f835a159-e668-421c-b478-2a7d9761aa01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:56:56.263969 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.263938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f835a159-e668-421c-b478-2a7d9761aa01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f835a159-e668-421c-b478-2a7d9761aa01" (UID: "f835a159-e668-421c-b478-2a7d9761aa01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:56:56.264056 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.263967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f835a159-e668-421c-b478-2a7d9761aa01-kube-api-access-87sh4" (OuterVolumeSpecName: "kube-api-access-87sh4") pod "f835a159-e668-421c-b478-2a7d9761aa01" (UID: "f835a159-e668-421c-b478-2a7d9761aa01"). InnerVolumeSpecName "kube-api-access-87sh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:56:56.268933 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.268839    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"]
Jul 16 17:56:56.363162 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.363118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca641694-2844-4e99-99f2-5c4f75d3eaec-proxy-tls\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.363324 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.363192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.363324 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.363217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2k2x\" (UniqueName: \"kubernetes.io/projected/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-api-access-f2k2x\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.363324 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.363295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca641694-2844-4e99-99f2-5c4f75d3eaec-kserve-provision-location\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.363453 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.363330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-cabundle-cert\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.363453 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.363391    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:56:56.363453 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.363408    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f835a159-e668-421c-b478-2a7d9761aa01-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:56:56.363453 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.363420    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f835a159-e668-421c-b478-2a7d9761aa01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:56:56.363453 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.363433    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f835a159-e668-421c-b478-2a7d9761aa01-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:56:56.363453 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.363447    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87sh4\" (UniqueName: \"kubernetes.io/projected/f835a159-e668-421c-b478-2a7d9761aa01-kube-api-access-87sh4\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:56:56.464496 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.464451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca641694-2844-4e99-99f2-5c4f75d3eaec-kserve-provision-location\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.464617 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.464496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-cabundle-cert\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.464617 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.464546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca641694-2844-4e99-99f2-5c4f75d3eaec-proxy-tls\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.464617 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.464610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.464747 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.464635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2k2x\" (UniqueName: \"kubernetes.io/projected/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-api-access-f2k2x\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.464747 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:56:56.464723    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-serving-cert: secret "isvc-init-fail-22eec6-predictor-serving-cert" not found
Jul 16 17:56:56.465020 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:56:56.464804    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca641694-2844-4e99-99f2-5c4f75d3eaec-proxy-tls podName:ca641694-2844-4e99-99f2-5c4f75d3eaec nodeName:}" failed. No retries permitted until 2026-07-16 17:56:56.964784527 +0000 UTC m=+1393.951320131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ca641694-2844-4e99-99f2-5c4f75d3eaec-proxy-tls") pod "isvc-init-fail-22eec6-predictor-584698c74b-2bmvv" (UID: "ca641694-2844-4e99-99f2-5c4f75d3eaec") : secret "isvc-init-fail-22eec6-predictor-serving-cert" not found
Jul 16 17:56:56.465020 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.464931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca641694-2844-4e99-99f2-5c4f75d3eaec-kserve-provision-location\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.465349 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.465325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-cabundle-cert\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.465386 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.465326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.473712 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.473684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2k2x\" (UniqueName: \"kubernetes.io/projected/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-api-access-f2k2x\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.591038 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.590947    2563 generic.go:356] "Generic (PLEG): container finished" podID="f835a159-e668-421c-b478-2a7d9761aa01" containerID="14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5" exitCode=0
Jul 16 17:56:56.591038 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.591002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" event={"ID":"f835a159-e668-421c-b478-2a7d9761aa01","Type":"ContainerDied","Data":"14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5"}
Jul 16 17:56:56.591231 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.591003    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"
Jul 16 17:56:56.591231 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.591059    2563 scope.go:117] "RemoveContainer" containerID="632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557"
Jul 16 17:56:56.591231 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.591048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks" event={"ID":"f835a159-e668-421c-b478-2a7d9761aa01","Type":"ContainerDied","Data":"92f46fc41b7478685059d2ef5e032b6ea4b4486964fddda4dfd9f8c795a1bad5"}
Jul 16 17:56:56.599691 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.599669    2563 scope.go:117] "RemoveContainer" containerID="14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5"
Jul 16 17:56:56.609709 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.609689    2563 scope.go:117] "RemoveContainer" containerID="28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647"
Jul 16 17:56:56.614326 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.614243    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"]
Jul 16 17:56:56.616312 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.616283    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-aae39d-predictor-d67696c6f-46tks"]
Jul 16 17:56:56.618487 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.618468    2563 scope.go:117] "RemoveContainer" containerID="632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557"
Jul 16 17:56:56.618828 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:56:56.618808    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557\": container with ID starting with 632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557 not found: ID does not exist" containerID="632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557"
Jul 16 17:56:56.618887 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.618840    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557"} err="failed to get container status \"632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557\": rpc error: code = NotFound desc = could not find container \"632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557\": container with ID starting with 632bbfa50913315431088e8cab4c413ab077ed8a216d66f403abaf8cdd5e2557 not found: ID does not exist"
Jul 16 17:56:56.618887 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.618861    2563 scope.go:117] "RemoveContainer" containerID="14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5"
Jul 16 17:56:56.619151 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:56:56.619129    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5\": container with ID starting with 14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5 not found: ID does not exist" containerID="14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5"
Jul 16 17:56:56.619227 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.619160    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5"} err="failed to get container status \"14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5\": rpc error: code = NotFound desc = could not find container \"14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5\": container with ID starting with 14cb3348c1db8a7e9111b75769b0fe680c3071a764cc6955c7f2eedf863893b5 not found: ID does not exist"
Jul 16 17:56:56.619227 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.619187    2563 scope.go:117] "RemoveContainer" containerID="28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647"
Jul 16 17:56:56.619433 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:56:56.619416    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647\": container with ID starting with 28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647 not found: ID does not exist" containerID="28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647"
Jul 16 17:56:56.619474 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.619438    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647"} err="failed to get container status \"28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647\": rpc error: code = NotFound desc = could not find container \"28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647\": container with ID starting with 28d1c700b2de4e1f0e63e471f5d819e89571a976ca8775c8d64c16caf315f647 not found: ID does not exist"
Jul 16 17:56:56.968170 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.968127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca641694-2844-4e99-99f2-5c4f75d3eaec-proxy-tls\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:56.970634 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:56.970610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca641694-2844-4e99-99f2-5c4f75d3eaec-proxy-tls\") pod \"isvc-init-fail-22eec6-predictor-584698c74b-2bmvv\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") " pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:57.164207 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:57.164160    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:56:57.307191 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:57.307164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"]
Jul 16 17:56:57.309440 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:56:57.309412    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca641694_2844_4e99_99f2_5c4f75d3eaec.slice/crio-a6f89fbcbe4a0dc9ffae47539ea2eebaea69019cba25064aa53169542c5a33b9 WatchSource:0}: Error finding container a6f89fbcbe4a0dc9ffae47539ea2eebaea69019cba25064aa53169542c5a33b9: Status 404 returned error can't find the container with id a6f89fbcbe4a0dc9ffae47539ea2eebaea69019cba25064aa53169542c5a33b9
Jul 16 17:56:57.596584 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:57.596459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv" event={"ID":"ca641694-2844-4e99-99f2-5c4f75d3eaec","Type":"ContainerStarted","Data":"d9139a7b585e10999b7b6875776f746af1a30acbc0c479a38cebd1f23430bd88"}
Jul 16 17:56:57.596584 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:57.596574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv" event={"ID":"ca641694-2844-4e99-99f2-5c4f75d3eaec","Type":"ContainerStarted","Data":"a6f89fbcbe4a0dc9ffae47539ea2eebaea69019cba25064aa53169542c5a33b9"}
Jul 16 17:56:57.674110 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:56:57.674077    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f835a159-e668-421c-b478-2a7d9761aa01" path="/var/lib/kubelet/pods/f835a159-e668-421c-b478-2a7d9761aa01/volumes"
Jul 16 17:57:01.616517 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:01.616479    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-22eec6-predictor-584698c74b-2bmvv_ca641694-2844-4e99-99f2-5c4f75d3eaec/storage-initializer/0.log"
Jul 16 17:57:01.616918 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:01.616527    2563 generic.go:356] "Generic (PLEG): container finished" podID="ca641694-2844-4e99-99f2-5c4f75d3eaec" containerID="d9139a7b585e10999b7b6875776f746af1a30acbc0c479a38cebd1f23430bd88" exitCode=1
Jul 16 17:57:01.616918 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:01.616578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv" event={"ID":"ca641694-2844-4e99-99f2-5c4f75d3eaec","Type":"ContainerDied","Data":"d9139a7b585e10999b7b6875776f746af1a30acbc0c479a38cebd1f23430bd88"}
Jul 16 17:57:02.622920 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:02.622893    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-22eec6-predictor-584698c74b-2bmvv_ca641694-2844-4e99-99f2-5c4f75d3eaec/storage-initializer/0.log"
Jul 16 17:57:02.623308 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:02.622944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv" event={"ID":"ca641694-2844-4e99-99f2-5c4f75d3eaec","Type":"ContainerStarted","Data":"8533cd549213d3cb31e0d546c8269265790e3f9112998b17d304948710174709"}
Jul 16 17:57:05.638190 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:05.638160    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-22eec6-predictor-584698c74b-2bmvv_ca641694-2844-4e99-99f2-5c4f75d3eaec/storage-initializer/1.log"
Jul 16 17:57:05.638637 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:05.638542    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-22eec6-predictor-584698c74b-2bmvv_ca641694-2844-4e99-99f2-5c4f75d3eaec/storage-initializer/0.log"
Jul 16 17:57:05.638637 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:05.638603    2563 generic.go:356] "Generic (PLEG): container finished" podID="ca641694-2844-4e99-99f2-5c4f75d3eaec" containerID="8533cd549213d3cb31e0d546c8269265790e3f9112998b17d304948710174709" exitCode=1
Jul 16 17:57:05.638719 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:05.638641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv" event={"ID":"ca641694-2844-4e99-99f2-5c4f75d3eaec","Type":"ContainerDied","Data":"8533cd549213d3cb31e0d546c8269265790e3f9112998b17d304948710174709"}
Jul 16 17:57:05.638719 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:05.638706    2563 scope.go:117] "RemoveContainer" containerID="d9139a7b585e10999b7b6875776f746af1a30acbc0c479a38cebd1f23430bd88"
Jul 16 17:57:05.639078 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:05.639060    2563 scope.go:117] "RemoveContainer" containerID="d9139a7b585e10999b7b6875776f746af1a30acbc0c479a38cebd1f23430bd88"
Jul 16 17:57:05.649934 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:57:05.649885    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-22eec6-predictor-584698c74b-2bmvv_kserve-ci-e2e-test_ca641694-2844-4e99-99f2-5c4f75d3eaec_0 in pod sandbox a6f89fbcbe4a0dc9ffae47539ea2eebaea69019cba25064aa53169542c5a33b9 from index: no such id: 'd9139a7b585e10999b7b6875776f746af1a30acbc0c479a38cebd1f23430bd88'" containerID="d9139a7b585e10999b7b6875776f746af1a30acbc0c479a38cebd1f23430bd88"
Jul 16 17:57:05.649934 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:57:05.649920    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-22eec6-predictor-584698c74b-2bmvv_kserve-ci-e2e-test(ca641694-2844-4e99-99f2-5c4f75d3eaec)\"" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv" podUID="ca641694-2844-4e99-99f2-5c4f75d3eaec"
Jul 16 17:57:05.650073 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:05.649941    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9139a7b585e10999b7b6875776f746af1a30acbc0c479a38cebd1f23430bd88"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-22eec6-predictor-584698c74b-2bmvv_kserve-ci-e2e-test_ca641694-2844-4e99-99f2-5c4f75d3eaec_0 in pod sandbox a6f89fbcbe4a0dc9ffae47539ea2eebaea69019cba25064aa53169542c5a33b9 from index: no such id: 'd9139a7b585e10999b7b6875776f746af1a30acbc0c479a38cebd1f23430bd88'"
Jul 16 17:57:06.268396 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.268358    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"]
Jul 16 17:57:06.644801 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.644719    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-22eec6-predictor-584698c74b-2bmvv_ca641694-2844-4e99-99f2-5c4f75d3eaec/storage-initializer/1.log"
Jul 16 17:57:06.778100 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.778074    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-22eec6-predictor-584698c74b-2bmvv_ca641694-2844-4e99-99f2-5c4f75d3eaec/storage-initializer/1.log"
Jul 16 17:57:06.781743 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.781717    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:57:06.851890 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.851851    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2k2x\" (UniqueName: \"kubernetes.io/projected/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-api-access-f2k2x\") pod \"ca641694-2844-4e99-99f2-5c4f75d3eaec\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") "
Jul 16 17:57:06.851890 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.851895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-rbac-proxy-sar-config\") pod \"ca641694-2844-4e99-99f2-5c4f75d3eaec\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") "
Jul 16 17:57:06.852131 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.851919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca641694-2844-4e99-99f2-5c4f75d3eaec-proxy-tls\") pod \"ca641694-2844-4e99-99f2-5c4f75d3eaec\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") "
Jul 16 17:57:06.852131 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.851945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca641694-2844-4e99-99f2-5c4f75d3eaec-kserve-provision-location\") pod \"ca641694-2844-4e99-99f2-5c4f75d3eaec\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") "
Jul 16 17:57:06.852131 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.851991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-cabundle-cert\") pod \"ca641694-2844-4e99-99f2-5c4f75d3eaec\" (UID: \"ca641694-2844-4e99-99f2-5c4f75d3eaec\") "
Jul 16 17:57:06.852292 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.852259    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca641694-2844-4e99-99f2-5c4f75d3eaec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca641694-2844-4e99-99f2-5c4f75d3eaec" (UID: "ca641694-2844-4e99-99f2-5c4f75d3eaec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:06.852345 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.852318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca641694-2844-4e99-99f2-5c4f75d3eaec" (UID: "ca641694-2844-4e99-99f2-5c4f75d3eaec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:57:06.852467 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.852445    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca641694-2844-4e99-99f2-5c4f75d3eaec" (UID: "ca641694-2844-4e99-99f2-5c4f75d3eaec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:57:06.854104 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.854078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-api-access-f2k2x" (OuterVolumeSpecName: "kube-api-access-f2k2x") pod "ca641694-2844-4e99-99f2-5c4f75d3eaec" (UID: "ca641694-2844-4e99-99f2-5c4f75d3eaec"). InnerVolumeSpecName "kube-api-access-f2k2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:57:06.854104 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.854095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca641694-2844-4e99-99f2-5c4f75d3eaec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca641694-2844-4e99-99f2-5c4f75d3eaec" (UID: "ca641694-2844-4e99-99f2-5c4f75d3eaec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:57:06.952736 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.952632    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2k2x\" (UniqueName: \"kubernetes.io/projected/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-api-access-f2k2x\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:57:06.952736 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.952667    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:57:06.952736 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.952684    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca641694-2844-4e99-99f2-5c4f75d3eaec-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:57:06.952736 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.952695    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca641694-2844-4e99-99f2-5c4f75d3eaec-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:57:06.952736 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:06.952709    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca641694-2844-4e99-99f2-5c4f75d3eaec-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:57:07.650407 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:07.650377    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-22eec6-predictor-584698c74b-2bmvv_ca641694-2844-4e99-99f2-5c4f75d3eaec/storage-initializer/1.log"
Jul 16 17:57:07.650931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:07.650493    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"
Jul 16 17:57:07.650931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:07.650491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv" event={"ID":"ca641694-2844-4e99-99f2-5c4f75d3eaec","Type":"ContainerDied","Data":"a6f89fbcbe4a0dc9ffae47539ea2eebaea69019cba25064aa53169542c5a33b9"}
Jul 16 17:57:07.650931 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:07.650626    2563 scope.go:117] "RemoveContainer" containerID="8533cd549213d3cb31e0d546c8269265790e3f9112998b17d304948710174709"
Jul 16 17:57:07.690697 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:07.690662    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"]
Jul 16 17:57:07.695172 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:07.695139    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-22eec6-predictor-584698c74b-2bmvv"]
Jul 16 17:57:09.672962 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:09.672923    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca641694-2844-4e99-99f2-5c4f75d3eaec" path="/var/lib/kubelet/pods/ca641694-2844-4e99-99f2-5c4f75d3eaec/volumes"
Jul 16 17:57:11.488384 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.488351    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"]
Jul 16 17:57:11.488890 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.488696    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca641694-2844-4e99-99f2-5c4f75d3eaec" containerName="storage-initializer"
Jul 16 17:57:11.488890 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.488709    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca641694-2844-4e99-99f2-5c4f75d3eaec" containerName="storage-initializer"
Jul 16 17:57:11.488890 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.488720    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca641694-2844-4e99-99f2-5c4f75d3eaec" containerName="storage-initializer"
Jul 16 17:57:11.488890 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.488726    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca641694-2844-4e99-99f2-5c4f75d3eaec" containerName="storage-initializer"
Jul 16 17:57:11.488890 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.488788    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca641694-2844-4e99-99f2-5c4f75d3eaec" containerName="storage-initializer"
Jul 16 17:57:11.488890 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.488888    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca641694-2844-4e99-99f2-5c4f75d3eaec" containerName="storage-initializer"
Jul 16 17:57:11.493383 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.493361    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.496646 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.496620    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 17:57:11.496810 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.496653    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:57:11.497041 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.497020    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-afd96-kube-rbac-proxy-sar-config\""
Jul 16 17:57:11.497312 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.497298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ft4dd\""
Jul 16 17:57:11.497312 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.497308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:57:11.497861 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.497845    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-afd96-predictor-serving-cert\""
Jul 16 17:57:11.497946 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.497912    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:57:11.504580 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.504536    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"]
Jul 16 17:57:11.592957 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.592914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szl6l\" (UniqueName: \"kubernetes.io/projected/69e56822-1430-4da0-85d0-630c33325a07-kube-api-access-szl6l\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.593158 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.592978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-cabundle-cert\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.593158 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.593020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e56822-1430-4da0-85d0-630c33325a07-proxy-tls\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.593158 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.593071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.593158 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.593088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69e56822-1430-4da0-85d0-630c33325a07-kserve-provision-location\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.693909 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.693870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.693909 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.693911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69e56822-1430-4da0-85d0-630c33325a07-kserve-provision-location\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.694145 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.693938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szl6l\" (UniqueName: \"kubernetes.io/projected/69e56822-1430-4da0-85d0-630c33325a07-kube-api-access-szl6l\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.694145 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.693975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-cabundle-cert\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.694145 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.693997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e56822-1430-4da0-85d0-630c33325a07-proxy-tls\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.694384 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.694357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69e56822-1430-4da0-85d0-630c33325a07-kserve-provision-location\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.694677 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.694647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-cabundle-cert\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.694788 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.694695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.696510 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.696478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e56822-1430-4da0-85d0-630c33325a07-proxy-tls\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.703987 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.703958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szl6l\" (UniqueName: \"kubernetes.io/projected/69e56822-1430-4da0-85d0-630c33325a07-kube-api-access-szl6l\") pod \"raw-sklearn-afd96-predictor-8d6f8647b-7hf9j\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") " pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.805081 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.804979    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:11.950608 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:11.950582    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"]
Jul 16 17:57:11.952595 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:57:11.952568    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e56822_1430_4da0_85d0_630c33325a07.slice/crio-5317a129b9cf9b9da6b8c9683f2e9b8dac66e9becb8b35ff458d75f8d0ff2e11 WatchSource:0}: Error finding container 5317a129b9cf9b9da6b8c9683f2e9b8dac66e9becb8b35ff458d75f8d0ff2e11: Status 404 returned error can't find the container with id 5317a129b9cf9b9da6b8c9683f2e9b8dac66e9becb8b35ff458d75f8d0ff2e11
Jul 16 17:57:12.676292 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:12.676249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" event={"ID":"69e56822-1430-4da0-85d0-630c33325a07","Type":"ContainerStarted","Data":"a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d"}
Jul 16 17:57:12.676711 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:12.676325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" event={"ID":"69e56822-1430-4da0-85d0-630c33325a07","Type":"ContainerStarted","Data":"5317a129b9cf9b9da6b8c9683f2e9b8dac66e9becb8b35ff458d75f8d0ff2e11"}
Jul 16 17:57:16.697288 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:16.697253    2563 generic.go:356] "Generic (PLEG): container finished" podID="69e56822-1430-4da0-85d0-630c33325a07" containerID="a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d" exitCode=0
Jul 16 17:57:16.697685 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:16.697328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" event={"ID":"69e56822-1430-4da0-85d0-630c33325a07","Type":"ContainerDied","Data":"a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d"}
Jul 16 17:57:17.703487 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:17.703453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" event={"ID":"69e56822-1430-4da0-85d0-630c33325a07","Type":"ContainerStarted","Data":"7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67"}
Jul 16 17:57:17.703879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:17.703515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" event={"ID":"69e56822-1430-4da0-85d0-630c33325a07","Type":"ContainerStarted","Data":"9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e"}
Jul 16 17:57:17.703879 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:17.703741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:17.726017 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:17.725963    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podStartSLOduration=2.903314657 podStartE2EDuration="6.725946496s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.822631839s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:17.724673654 +0000 UTC m=+1414.711209261" watchObservedRunningTime="2026-07-16 17:57:17.725946496 +0000 UTC m=+1414.712482101"
Jul 16 17:57:18.708532 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:18.708490    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:18.709885 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:18.709858    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 16 17:57:19.717124 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:19.717078    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 16 17:57:24.718873 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:24.718841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:57:24.719390 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:24.719358    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 16 17:57:34.720111 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:34.720064    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 16 17:57:44.719845 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:44.719805    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 16 17:57:54.720067 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:57:54.720026    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 16 17:58:04.719994 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:04.719944    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 16 17:58:14.719467 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:14.719421    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 16 17:58:24.719336 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:24.719286    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 16 17:58:34.720423 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:34.720395    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:58:41.609581 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:41.609533    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"]
Jul 16 17:58:41.610036 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:41.609874    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" containerID="cri-o://9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e" gracePeriod=30
Jul 16 17:58:41.610036 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:41.609923    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kube-rbac-proxy" containerID="cri-o://7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67" gracePeriod=30
Jul 16 17:58:42.066326 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:42.066294    2563 generic.go:356] "Generic (PLEG): container finished" podID="69e56822-1430-4da0-85d0-630c33325a07" containerID="7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67" exitCode=2
Jul 16 17:58:42.066498 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:42.066369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" event={"ID":"69e56822-1430-4da0-85d0-630c33325a07","Type":"ContainerDied","Data":"7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67"}
Jul 16 17:58:43.595259 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:43.595212    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:58:43.598290 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:43.598267    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:58:43.601850 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:43.601815    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 17:58:43.602672 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:43.602651    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:58:43.604688 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:43.604668    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 17:58:43.608534 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:43.608516    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 17:58:44.714962 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:44.714918    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 16 17:58:44.719716 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:44.719686    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 16 17:58:46.563405 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.563373    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:58:46.704755 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.704662    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szl6l\" (UniqueName: \"kubernetes.io/projected/69e56822-1430-4da0-85d0-630c33325a07-kube-api-access-szl6l\") pod \"69e56822-1430-4da0-85d0-630c33325a07\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") "
Jul 16 17:58:46.704755 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.704700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-cabundle-cert\") pod \"69e56822-1430-4da0-85d0-630c33325a07\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") "
Jul 16 17:58:46.704755 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.704737    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e56822-1430-4da0-85d0-630c33325a07-proxy-tls\") pod \"69e56822-1430-4da0-85d0-630c33325a07\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") "
Jul 16 17:58:46.704993 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.704770    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-kube-rbac-proxy-sar-config\") pod \"69e56822-1430-4da0-85d0-630c33325a07\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") "
Jul 16 17:58:46.704993 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.704835    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69e56822-1430-4da0-85d0-630c33325a07-kserve-provision-location\") pod \"69e56822-1430-4da0-85d0-630c33325a07\" (UID: \"69e56822-1430-4da0-85d0-630c33325a07\") "
Jul 16 17:58:46.705224 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.705170    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69e56822-1430-4da0-85d0-630c33325a07" (UID: "69e56822-1430-4da0-85d0-630c33325a07"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:58:46.705362 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.705277    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e56822-1430-4da0-85d0-630c33325a07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69e56822-1430-4da0-85d0-630c33325a07" (UID: "69e56822-1430-4da0-85d0-630c33325a07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:58:46.705460 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.705431    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69e56822-1430-4da0-85d0-630c33325a07" (UID: "69e56822-1430-4da0-85d0-630c33325a07"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:58:46.706907 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.706887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e56822-1430-4da0-85d0-630c33325a07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69e56822-1430-4da0-85d0-630c33325a07" (UID: "69e56822-1430-4da0-85d0-630c33325a07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:58:46.707142 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.707114    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e56822-1430-4da0-85d0-630c33325a07-kube-api-access-szl6l" (OuterVolumeSpecName: "kube-api-access-szl6l") pod "69e56822-1430-4da0-85d0-630c33325a07" (UID: "69e56822-1430-4da0-85d0-630c33325a07"). InnerVolumeSpecName "kube-api-access-szl6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:58:46.806014 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.805966    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szl6l\" (UniqueName: \"kubernetes.io/projected/69e56822-1430-4da0-85d0-630c33325a07-kube-api-access-szl6l\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:58:46.806014 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.806005    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:58:46.806014 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.806014    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e56822-1430-4da0-85d0-630c33325a07-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:58:46.806014 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.806023    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e56822-1430-4da0-85d0-630c33325a07-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:58:46.806478 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.806032    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69e56822-1430-4da0-85d0-630c33325a07-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 17:58:46.808695 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.808662    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"]
Jul 16 17:58:46.809013 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.809000    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="storage-initializer"
Jul 16 17:58:46.809068 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.809014    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="storage-initializer"
Jul 16 17:58:46.809068 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.809022    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kube-rbac-proxy"
Jul 16 17:58:46.809068 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.809028    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kube-rbac-proxy"
Jul 16 17:58:46.809068 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.809041    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container"
Jul 16 17:58:46.809068 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.809046    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container"
Jul 16 17:58:46.809222 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.809104    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kserve-container"
Jul 16 17:58:46.809222 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.809115    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e56822-1430-4da0-85d0-630c33325a07" containerName="kube-rbac-proxy"
Jul 16 17:58:46.812608 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.812586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:46.815307 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.815282    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-9159d-predictor-serving-cert\""
Jul 16 17:58:46.815443 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.815286    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-9159d-kube-rbac-proxy-sar-config\""
Jul 16 17:58:46.822251 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.822226    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"]
Jul 16 17:58:46.907217 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.907174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d8630e6-683e-4fe9-acad-7602420ec371-kserve-provision-location\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:46.907217 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.907216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-cabundle-cert\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:46.907435 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.907253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8630e6-683e-4fe9-acad-7602420ec371-proxy-tls\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:46.907435 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.907309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:46.907435 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:46.907328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n9hc\" (UniqueName: \"kubernetes.io/projected/9d8630e6-683e-4fe9-acad-7602420ec371-kube-api-access-5n9hc\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.008676 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.008626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8630e6-683e-4fe9-acad-7602420ec371-proxy-tls\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.008845 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.008712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.008845 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.008738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n9hc\" (UniqueName: \"kubernetes.io/projected/9d8630e6-683e-4fe9-acad-7602420ec371-kube-api-access-5n9hc\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.008845 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.008768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d8630e6-683e-4fe9-acad-7602420ec371-kserve-provision-location\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.008845 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.008786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-cabundle-cert\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.008845 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:58:47.008788    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-serving-cert: secret "raw-sklearn-runtime-9159d-predictor-serving-cert" not found
Jul 16 17:58:47.009062 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:58:47.008904    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8630e6-683e-4fe9-acad-7602420ec371-proxy-tls podName:9d8630e6-683e-4fe9-acad-7602420ec371 nodeName:}" failed. No retries permitted until 2026-07-16 17:58:47.508883756 +0000 UTC m=+1504.495419341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9d8630e6-683e-4fe9-acad-7602420ec371-proxy-tls") pod "raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" (UID: "9d8630e6-683e-4fe9-acad-7602420ec371") : secret "raw-sklearn-runtime-9159d-predictor-serving-cert" not found
Jul 16 17:58:47.009213 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.009187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d8630e6-683e-4fe9-acad-7602420ec371-kserve-provision-location\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.009404 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.009387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.009455 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.009438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-cabundle-cert\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.022263 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.022236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n9hc\" (UniqueName: \"kubernetes.io/projected/9d8630e6-683e-4fe9-acad-7602420ec371-kube-api-access-5n9hc\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.088185 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.088152    2563 generic.go:356] "Generic (PLEG): container finished" podID="69e56822-1430-4da0-85d0-630c33325a07" containerID="9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e" exitCode=0
Jul 16 17:58:47.088366 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.088219    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"
Jul 16 17:58:47.088366 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.088231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" event={"ID":"69e56822-1430-4da0-85d0-630c33325a07","Type":"ContainerDied","Data":"9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e"}
Jul 16 17:58:47.088366 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.088296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j" event={"ID":"69e56822-1430-4da0-85d0-630c33325a07","Type":"ContainerDied","Data":"5317a129b9cf9b9da6b8c9683f2e9b8dac66e9becb8b35ff458d75f8d0ff2e11"}
Jul 16 17:58:47.088366 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.088322    2563 scope.go:117] "RemoveContainer" containerID="7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67"
Jul 16 17:58:47.097285 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.097264    2563 scope.go:117] "RemoveContainer" containerID="9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e"
Jul 16 17:58:47.105430 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.105410    2563 scope.go:117] "RemoveContainer" containerID="a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d"
Jul 16 17:58:47.111663 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.111632    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"]
Jul 16 17:58:47.113237 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.113178    2563 scope.go:117] "RemoveContainer" containerID="7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67"
Jul 16 17:58:47.113634 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:58:47.113602    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67\": container with ID starting with 7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67 not found: ID does not exist" containerID="7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67"
Jul 16 17:58:47.113740 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.113639    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67"} err="failed to get container status \"7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67\": rpc error: code = NotFound desc = could not find container \"7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67\": container with ID starting with 7e09712f9d4bc082d02681de6e04fa77ceb97ca9b89ae5fb86187c57c0fd4c67 not found: ID does not exist"
Jul 16 17:58:47.113740 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.113669    2563 scope.go:117] "RemoveContainer" containerID="9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e"
Jul 16 17:58:47.114087 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:58:47.114065    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e\": container with ID starting with 9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e not found: ID does not exist" containerID="9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e"
Jul 16 17:58:47.114167 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.114091    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e"} err="failed to get container status \"9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e\": rpc error: code = NotFound desc = could not find container \"9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e\": container with ID starting with 9dde586e81bc11450cbb19d47ecabde931ff8bf30956986d6bbce9c19868065e not found: ID does not exist"
Jul 16 17:58:47.114167 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.114108    2563 scope.go:117] "RemoveContainer" containerID="a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d"
Jul 16 17:58:47.114386 ip-10-0-132-218 kubenswrapper[2563]: E0716 17:58:47.114370    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d\": container with ID starting with a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d not found: ID does not exist" containerID="a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d"
Jul 16 17:58:47.114444 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.114388    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d"} err="failed to get container status \"a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d\": rpc error: code = NotFound desc = could not find container \"a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d\": container with ID starting with a710f655724616535aade26c384ad324a7ea85ab3eca2bbd07634f392bab314d not found: ID does not exist"
Jul 16 17:58:47.115517 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.115495    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-afd96-predictor-8d6f8647b-7hf9j"]
Jul 16 17:58:47.513608 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.513544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8630e6-683e-4fe9-acad-7602420ec371-proxy-tls\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.515925 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.515904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8630e6-683e-4fe9-acad-7602420ec371-proxy-tls\") pod \"raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.673110 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.673072    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69e56822-1430-4da0-85d0-630c33325a07" path="/var/lib/kubelet/pods/69e56822-1430-4da0-85d0-630c33325a07/volumes"
Jul 16 17:58:47.724037 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.724007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:47.861969 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:47.861942    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"]
Jul 16 17:58:47.863943 ip-10-0-132-218 kubenswrapper[2563]: W0716 17:58:47.863914    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8630e6_683e_4fe9_acad_7602420ec371.slice/crio-adf706f733af91364c23641501775a896cced8ad0be21e75c75b9d24a9e16a41 WatchSource:0}: Error finding container adf706f733af91364c23641501775a896cced8ad0be21e75c75b9d24a9e16a41: Status 404 returned error can't find the container with id adf706f733af91364c23641501775a896cced8ad0be21e75c75b9d24a9e16a41
Jul 16 17:58:48.096469 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:48.096371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" event={"ID":"9d8630e6-683e-4fe9-acad-7602420ec371","Type":"ContainerStarted","Data":"79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b"}
Jul 16 17:58:48.096469 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:48.096430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" event={"ID":"9d8630e6-683e-4fe9-acad-7602420ec371","Type":"ContainerStarted","Data":"adf706f733af91364c23641501775a896cced8ad0be21e75c75b9d24a9e16a41"}
Jul 16 17:58:52.115347 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:52.115258    2563 generic.go:356] "Generic (PLEG): container finished" podID="9d8630e6-683e-4fe9-acad-7602420ec371" containerID="79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b" exitCode=0
Jul 16 17:58:52.115347 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:52.115332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" event={"ID":"9d8630e6-683e-4fe9-acad-7602420ec371","Type":"ContainerDied","Data":"79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b"}
Jul 16 17:58:53.121435 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:53.121401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" event={"ID":"9d8630e6-683e-4fe9-acad-7602420ec371","Type":"ContainerStarted","Data":"3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444"}
Jul 16 17:58:53.121834 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:53.121469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" event={"ID":"9d8630e6-683e-4fe9-acad-7602420ec371","Type":"ContainerStarted","Data":"fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef"}
Jul 16 17:58:53.121834 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:53.121684    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:54.125782 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:54.125750    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:58:54.127031 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:54.126998    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 16 17:58:55.130190 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:58:55.130146    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 16 17:59:00.135204 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:59:00.135174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 17:59:00.135819 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:59:00.135790    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 16 17:59:00.156825 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:59:00.156769    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podStartSLOduration=10.359756333 podStartE2EDuration="14.156754014s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.796997681s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:58:53.148242994 +0000 UTC m=+1510.134778601" watchObservedRunningTime="2026-07-16 17:59:00.156754014 +0000 UTC m=+1517.143289621"
Jul 16 17:59:10.136705 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:59:10.136663    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 16 17:59:20.136384 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:59:20.136341    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 16 17:59:30.136687 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:59:30.136647    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 16 17:59:40.136244 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:59:40.136203    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 16 17:59:50.136391 ip-10-0-132-218 kubenswrapper[2563]: I0716 17:59:50.136346    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 16 18:00:00.136216 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:00.136177    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 16 18:00:03.670935 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:03.670893    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 16 18:00:13.678805 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:13.673593    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 18:00:16.970053 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:16.970021    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"]
Jul 16 18:00:16.970533 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:16.970328    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container" containerID="cri-o://fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef" gracePeriod=30
Jul 16 18:00:16.970533 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:16.970340    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kube-rbac-proxy" containerID="cri-o://3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444" gracePeriod=30
Jul 16 18:00:17.469134 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:17.469099    2563 generic.go:356] "Generic (PLEG): container finished" podID="9d8630e6-683e-4fe9-acad-7602420ec371" containerID="3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444" exitCode=2
Jul 16 18:00:17.469295 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:17.469167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" event={"ID":"9d8630e6-683e-4fe9-acad-7602420ec371","Type":"ContainerDied","Data":"3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444"}
Jul 16 18:00:20.130740 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:20.130692    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 16 18:00:22.019215 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.019186    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 18:00:22.113507 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.113404    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-kube-rbac-proxy-sar-config\") pod \"9d8630e6-683e-4fe9-acad-7602420ec371\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") "
Jul 16 18:00:22.113507 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.113449    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d8630e6-683e-4fe9-acad-7602420ec371-kserve-provision-location\") pod \"9d8630e6-683e-4fe9-acad-7602420ec371\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") "
Jul 16 18:00:22.113507 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.113478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n9hc\" (UniqueName: \"kubernetes.io/projected/9d8630e6-683e-4fe9-acad-7602420ec371-kube-api-access-5n9hc\") pod \"9d8630e6-683e-4fe9-acad-7602420ec371\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") "
Jul 16 18:00:22.113507 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.113508    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-cabundle-cert\") pod \"9d8630e6-683e-4fe9-acad-7602420ec371\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") "
Jul 16 18:00:22.113861 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.113592    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8630e6-683e-4fe9-acad-7602420ec371-proxy-tls\") pod \"9d8630e6-683e-4fe9-acad-7602420ec371\" (UID: \"9d8630e6-683e-4fe9-acad-7602420ec371\") "
Jul 16 18:00:22.113861 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.113822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8630e6-683e-4fe9-acad-7602420ec371-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d8630e6-683e-4fe9-acad-7602420ec371" (UID: "9d8630e6-683e-4fe9-acad-7602420ec371"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:00:22.113974 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.113862    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d8630e6-683e-4fe9-acad-7602420ec371" (UID: "9d8630e6-683e-4fe9-acad-7602420ec371"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:00:22.113974 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.113866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d8630e6-683e-4fe9-acad-7602420ec371" (UID: "9d8630e6-683e-4fe9-acad-7602420ec371"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:00:22.115619 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.115592    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8630e6-683e-4fe9-acad-7602420ec371-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d8630e6-683e-4fe9-acad-7602420ec371" (UID: "9d8630e6-683e-4fe9-acad-7602420ec371"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:00:22.115734 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.115650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8630e6-683e-4fe9-acad-7602420ec371-kube-api-access-5n9hc" (OuterVolumeSpecName: "kube-api-access-5n9hc") pod "9d8630e6-683e-4fe9-acad-7602420ec371" (UID: "9d8630e6-683e-4fe9-acad-7602420ec371"). InnerVolumeSpecName "kube-api-access-5n9hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:00:22.214439 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.214398    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d8630e6-683e-4fe9-acad-7602420ec371-kserve-provision-location\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 18:00:22.214439 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.214434    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n9hc\" (UniqueName: \"kubernetes.io/projected/9d8630e6-683e-4fe9-acad-7602420ec371-kube-api-access-5n9hc\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 18:00:22.214439 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.214443    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-cabundle-cert\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 18:00:22.214439 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.214453    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8630e6-683e-4fe9-acad-7602420ec371-proxy-tls\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 18:00:22.214758 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.214461    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d8630e6-683e-4fe9-acad-7602420ec371-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-218.ec2.internal\" DevicePath \"\""
Jul 16 18:00:22.490993 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.490960    2563 generic.go:356] "Generic (PLEG): container finished" podID="9d8630e6-683e-4fe9-acad-7602420ec371" containerID="fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef" exitCode=0
Jul 16 18:00:22.491164 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.491027    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"
Jul 16 18:00:22.491164 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.491030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" event={"ID":"9d8630e6-683e-4fe9-acad-7602420ec371","Type":"ContainerDied","Data":"fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef"}
Jul 16 18:00:22.491164 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.491092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls" event={"ID":"9d8630e6-683e-4fe9-acad-7602420ec371","Type":"ContainerDied","Data":"adf706f733af91364c23641501775a896cced8ad0be21e75c75b9d24a9e16a41"}
Jul 16 18:00:22.491164 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.491132    2563 scope.go:117] "RemoveContainer" containerID="3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444"
Jul 16 18:00:22.499984 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.499960    2563 scope.go:117] "RemoveContainer" containerID="fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef"
Jul 16 18:00:22.508074 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.508054    2563 scope.go:117] "RemoveContainer" containerID="79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b"
Jul 16 18:00:22.515122 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.515092    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"]
Jul 16 18:00:22.516363 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.516346    2563 scope.go:117] "RemoveContainer" containerID="3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444"
Jul 16 18:00:22.516682 ip-10-0-132-218 kubenswrapper[2563]: E0716 18:00:22.516662    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444\": container with ID starting with 3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444 not found: ID does not exist" containerID="3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444"
Jul 16 18:00:22.516736 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.516692    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444"} err="failed to get container status \"3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444\": rpc error: code = NotFound desc = could not find container \"3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444\": container with ID starting with 3732c6eae665511f953b61cc3184568a9b06ee5e71789780f529de6039416444 not found: ID does not exist"
Jul 16 18:00:22.516736 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.516712    2563 scope.go:117] "RemoveContainer" containerID="fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef"
Jul 16 18:00:22.516968 ip-10-0-132-218 kubenswrapper[2563]: E0716 18:00:22.516951    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef\": container with ID starting with fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef not found: ID does not exist" containerID="fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef"
Jul 16 18:00:22.517012 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.516974    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef"} err="failed to get container status \"fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef\": rpc error: code = NotFound desc = could not find container \"fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef\": container with ID starting with fa5c72cca7ddb5c46cdc14568b3db34f09368e8fc27401f65378149fe794cfef not found: ID does not exist"
Jul 16 18:00:22.517012 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.516996    2563 scope.go:117] "RemoveContainer" containerID="79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b"
Jul 16 18:00:22.517238 ip-10-0-132-218 kubenswrapper[2563]: E0716 18:00:22.517219    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b\": container with ID starting with 79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b not found: ID does not exist" containerID="79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b"
Jul 16 18:00:22.517289 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.517243    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b"} err="failed to get container status \"79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b\": rpc error: code = NotFound desc = could not find container \"79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b\": container with ID starting with 79c2a5def889c184a1894dc7b8d48b560bc5ec84a2692facb44b7c540971ae1b not found: ID does not exist"
Jul 16 18:00:22.520620 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:22.520600    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9159d-predictor-bf65d4648-cbjls"]
Jul 16 18:00:23.673661 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:23.673627    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" path="/var/lib/kubelet/pods/9d8630e6-683e-4fe9-acad-7602420ec371/volumes"
Jul 16 18:00:56.937666 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:56.937634    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tnzn7_ef3ddfba-d14b-493c-8c03-0c96efeb611b/global-pull-secret-syncer/0.log"
Jul 16 18:00:57.056443 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:57.056408    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rsfgv_d656ee83-7adb-4ed3-8255-215c6736b17a/konnectivity-agent/0.log"
Jul 16 18:00:57.105696 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:00:57.105664    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-218.ec2.internal_2b7aea08f316a1dd14e58ba5cd1a755a/haproxy/0.log"
Jul 16 18:01:01.051209 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:01.051167    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/1.log"
Jul 16 18:01:01.125494 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:01.125464    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v4b27_54fdfddd-0dd4-4af2-90bc-ccaac4996484/cluster-monitoring-operator/0.log"
Jul 16 18:01:01.339789 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:01.339762    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-69lnj_657cd3e3-3c63-4709-a8ed-885d35227bfa/node-exporter/0.log"
Jul 16 18:01:01.373535 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:01.373511    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-69lnj_657cd3e3-3c63-4709-a8ed-885d35227bfa/kube-rbac-proxy/0.log"
Jul 16 18:01:01.398784 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:01.398758    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-69lnj_657cd3e3-3c63-4709-a8ed-885d35227bfa/init-textfile/0.log"
Jul 16 18:01:01.893026 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:01.892996    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-ljk48_5b39b98b-0c71-4664-ac40-b0a6b4ce520e/prometheus-operator/0.log"
Jul 16 18:01:01.917865 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:01.917838    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-ljk48_5b39b98b-0c71-4664-ac40-b0a6b4ce520e/kube-rbac-proxy/0.log"
Jul 16 18:01:01.981317 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:01.981290    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cf68d7f88-6vtrb_0ea7c990-25ec-4785-bf6e-4f5e5f2cc847/telemeter-client/0.log"
Jul 16 18:01:02.004758 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:02.004736    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cf68d7f88-6vtrb_0ea7c990-25ec-4785-bf6e-4f5e5f2cc847/reload/0.log"
Jul 16 18:01:02.030003 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:02.029968    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cf68d7f88-6vtrb_0ea7c990-25ec-4785-bf6e-4f5e5f2cc847/kube-rbac-proxy/0.log"
Jul 16 18:01:02.059384 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:02.059359    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ccd95c578-xn2c9_b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa/thanos-query/0.log"
Jul 16 18:01:02.084957 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:02.084931    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ccd95c578-xn2c9_b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa/kube-rbac-proxy-web/0.log"
Jul 16 18:01:02.107848 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:02.107823    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ccd95c578-xn2c9_b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa/kube-rbac-proxy/0.log"
Jul 16 18:01:02.128038 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:02.128005    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ccd95c578-xn2c9_b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa/prom-label-proxy/0.log"
Jul 16 18:01:02.150647 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:02.150573    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ccd95c578-xn2c9_b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa/kube-rbac-proxy-rules/0.log"
Jul 16 18:01:02.173862 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:02.173837    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ccd95c578-xn2c9_b9f2e90e-67ed-4fe1-bbaf-9b15e2ba3bfa/kube-rbac-proxy-metrics/0.log"
Jul 16 18:01:04.018887 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:04.018857    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/1.log"
Jul 16 18:01:04.026341 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:04.026308    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6dgms_b2c17e8e-e654-40b4-84a6-6685fcd6e3cb/console-operator/2.log"
Jul 16 18:01:04.490937 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:04.490907    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-cr7t9_bf23d9f0-60cd-49e2-b943-098589373c45/download-server/0.log"
Jul 16 18:01:05.461533 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.461498    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"]
Jul 16 18:01:05.462034 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.462010    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container"
Jul 16 18:01:05.462034 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.462030    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container"
Jul 16 18:01:05.462149 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.462047    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kube-rbac-proxy"
Jul 16 18:01:05.462149 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.462055    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kube-rbac-proxy"
Jul 16 18:01:05.462149 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.462076    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="storage-initializer"
Jul 16 18:01:05.462149 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.462086    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="storage-initializer"
Jul 16 18:01:05.462418 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.462178    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kube-rbac-proxy"
Jul 16 18:01:05.462418 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.462191    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8630e6-683e-4fe9-acad-7602420ec371" containerName="kserve-container"
Jul 16 18:01:05.465328 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.465308    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.468466 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.468439    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trlt4\"/\"kube-root-ca.crt\""
Jul 16 18:01:05.469770 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.469753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-trlt4\"/\"default-dockercfg-qc85p\""
Jul 16 18:01:05.469870 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.469770    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trlt4\"/\"openshift-service-ca.crt\""
Jul 16 18:01:05.473106 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.473087    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"]
Jul 16 18:01:05.571216 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.571177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-podres\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.571216 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.571224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-sys\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.571482 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.571266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-lib-modules\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.571482 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.571323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz2z5\" (UniqueName: \"kubernetes.io/projected/7779561f-4e7d-44dd-b3a9-acd232d2ad50-kube-api-access-fz2z5\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.571482 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.571371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-proc\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.672515 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.672473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-lib-modules\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.672515 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.672518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2z5\" (UniqueName: \"kubernetes.io/projected/7779561f-4e7d-44dd-b3a9-acd232d2ad50-kube-api-access-fz2z5\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.672778 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.672561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-proc\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.672778 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.672624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-podres\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.672778 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.672657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-sys\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.672778 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.672659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-lib-modules\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.672778 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.672703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-proc\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.672778 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.672729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-sys\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.672778 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.672729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7779561f-4e7d-44dd-b3a9-acd232d2ad50-podres\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.681840 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.681818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2z5\" (UniqueName: \"kubernetes.io/projected/7779561f-4e7d-44dd-b3a9-acd232d2ad50-kube-api-access-fz2z5\") pod \"perf-node-gather-daemonset-875c5\" (UID: \"7779561f-4e7d-44dd-b3a9-acd232d2ad50\") " pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.775934 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.775897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:05.910463 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:05.910436    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"]
Jul 16 18:01:05.912793 ip-10-0-132-218 kubenswrapper[2563]: W0716 18:01:05.912756    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7779561f_4e7d_44dd_b3a9_acd232d2ad50.slice/crio-cf9aa92cb070e923a5dfc9d79f97292663b1a85b1129160dd15e86da47f4ddbc WatchSource:0}: Error finding container cf9aa92cb070e923a5dfc9d79f97292663b1a85b1129160dd15e86da47f4ddbc: Status 404 returned error can't find the container with id cf9aa92cb070e923a5dfc9d79f97292663b1a85b1129160dd15e86da47f4ddbc
Jul 16 18:01:06.030032 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:06.029944    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nqwmf_b1836785-46a4-4231-a977-eee3a2d8b563/dns/0.log"
Jul 16 18:01:06.049979 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:06.049951    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nqwmf_b1836785-46a4-4231-a977-eee3a2d8b563/kube-rbac-proxy/0.log"
Jul 16 18:01:06.171011 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:06.170977    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mxzqv_fa1f394b-ac23-4640-86ed-75f04466fa1e/dns-node-resolver/0.log"
Jul 16 18:01:06.637157 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:06.637128    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-869b4d4848-dpsm9_93efcccf-8ddb-461e-a1cd-a9a901480513/registry/0.log"
Jul 16 18:01:06.673686 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:06.673656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5" event={"ID":"7779561f-4e7d-44dd-b3a9-acd232d2ad50","Type":"ContainerStarted","Data":"0f54925c6c5c5131c82567ff02306825c900b32ee62639896a577538808b36bd"}
Jul 16 18:01:06.673686 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:06.673700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5" event={"ID":"7779561f-4e7d-44dd-b3a9-acd232d2ad50","Type":"ContainerStarted","Data":"cf9aa92cb070e923a5dfc9d79f97292663b1a85b1129160dd15e86da47f4ddbc"}
Jul 16 18:01:06.673930 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:06.673748    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:06.695250 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:06.695199    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5" podStartSLOduration=1.695182016 podStartE2EDuration="1.695182016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:01:06.692724859 +0000 UTC m=+1643.679260467" watchObservedRunningTime="2026-07-16 18:01:06.695182016 +0000 UTC m=+1643.681717625"
Jul 16 18:01:06.717563 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:06.717521    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vdpc9_04ee78ee-1c82-4394-8b56-23cda890ae80/node-ca/0.log"
Jul 16 18:01:07.631182 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:07.631156    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cb66c8d8-nn55n_5569cfbb-8a85-4f59-b995-32f0a18d7857/router/0.log"
Jul 16 18:01:08.039745 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:08.039719    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6z8w2_fd249ea5-c446-40d1-97b1-97223cc47d84/serve-healthcheck-canary/0.log"
Jul 16 18:01:08.501100 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:08.501070    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qwdm_80ceb78a-1472-4ff4-889f-dc7a16520c1c/kube-rbac-proxy/0.log"
Jul 16 18:01:08.525911 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:08.525885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qwdm_80ceb78a-1472-4ff4-889f-dc7a16520c1c/exporter/0.log"
Jul 16 18:01:08.552160 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:08.552127    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qwdm_80ceb78a-1472-4ff4-889f-dc7a16520c1c/extractor/0.log"
Jul 16 18:01:10.801364 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:10.801332    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-jcfs9_bcc8784c-c8c8-427f-8af5-5d3f9ca28dd8/manager/0.log"
Jul 16 18:01:12.686476 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:12.686449    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-trlt4/perf-node-gather-daemonset-875c5"
Jul 16 18:01:16.634224 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:16.634135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sdp_dea026da-38bb-4bf7-9382-7405d0abc6d4/kube-multus-additional-cni-plugins/0.log"
Jul 16 18:01:16.657968 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:16.657942    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sdp_dea026da-38bb-4bf7-9382-7405d0abc6d4/egress-router-binary-copy/0.log"
Jul 16 18:01:16.681680 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:16.681654    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sdp_dea026da-38bb-4bf7-9382-7405d0abc6d4/cni-plugins/0.log"
Jul 16 18:01:16.716138 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:16.716110    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sdp_dea026da-38bb-4bf7-9382-7405d0abc6d4/bond-cni-plugin/0.log"
Jul 16 18:01:16.737237 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:16.737205    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sdp_dea026da-38bb-4bf7-9382-7405d0abc6d4/routeoverride-cni/0.log"
Jul 16 18:01:16.760350 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:16.760320    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sdp_dea026da-38bb-4bf7-9382-7405d0abc6d4/whereabouts-cni-bincopy/0.log"
Jul 16 18:01:16.782276 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:16.782247    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sdp_dea026da-38bb-4bf7-9382-7405d0abc6d4/whereabouts-cni/0.log"
Jul 16 18:01:17.175908 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:17.175876    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-whxjd_4a921cd4-90af-41b4-828f-9c74ce755d39/kube-multus/0.log"
Jul 16 18:01:17.308279 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:17.308249    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gn4pn_c928af6c-6ba7-40bb-8e7a-4a418754b963/network-metrics-daemon/0.log"
Jul 16 18:01:17.330130 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:17.330100    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gn4pn_c928af6c-6ba7-40bb-8e7a-4a418754b963/kube-rbac-proxy/0.log"
Jul 16 18:01:18.119432 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:18.119407    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-controller/0.log"
Jul 16 18:01:18.141155 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:18.141127    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/0.log"
Jul 16 18:01:18.141318 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:18.141164    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovn-acl-logging/1.log"
Jul 16 18:01:18.159114 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:18.159086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/kube-rbac-proxy-node/0.log"
Jul 16 18:01:18.181015 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:18.180992    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/kube-rbac-proxy-ovn-metrics/0.log"
Jul 16 18:01:18.202138 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:18.202113    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/northd/0.log"
Jul 16 18:01:18.223105 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:18.223077    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/nbdb/0.log"
Jul 16 18:01:18.246620 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:18.246565    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/sbdb/0.log"
Jul 16 18:01:18.287242 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:18.287208    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4ts_ffe1aeb1-9246-430c-964b-2eb6db9dd017/ovnkube-controller/0.log"
Jul 16 18:01:19.711611 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:19.711585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-hr44p_33b91431-5098-4f19-9e2f-489917539c2f/network-check-target-container/0.log"
Jul 16 18:01:20.669939 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:20.669910    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-wnvg9_2f7cb219-cff0-464c-b353-97537f8d29c3/iptables-alerter/0.log"
Jul 16 18:01:21.409302 ip-10-0-132-218 kubenswrapper[2563]: I0716 18:01:21.409230    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-949cg_63d63b93-8c90-401b-ba2c-1585ea70fb35/tuned/0.log"